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ABSTRACT 

A study was done to assess implementation of the 
Chapter 1 program improvement provisions of the Mawkins-Staf f ord 
Elementary and Secondary School Improvement Amendments of 1988. 
Chapter 1 is the largest federal education program for educationally 
deprived school children, and the Hawkins-Stafford Act was designed 
to ensure that individual schools are effective in helping students 
achieve Chapter 1 goals. Specifically, the study reviewed the process 
used to identify schools in need of program improvement and determine 
whether this process could be improved and to compare implementation 
of the joint and local phases of program improvement. The study used 
multiple methodologies including statistical analysis, mail surveys, 
case studies, and a literature review. The results show that the 
process used to identify schools for program improvements is flawed 
because schools often are evaluated only on whether their Chapter 1 
students show annual increases in achievement test scores. When 
schools are evaluated only on achievement test scores, many schools 
are likely to be judged on changes in test scores that reflect random 
fluctuations rather than actual changes in student performance. 
Chapter 1 policy does not adequately explain how evidence from 
multiple indicators of student performance may be interpreted. In 
addition, pressure to increase test scores may have negative effects 
on instruction. Includes 5 tables; 19 figures; 45 references; and 
extensive appendixes containing officials 1 views, technical and 
methodological information, survey results, U.S. Department of 
Education comments, and a list of major contributors to the report. 
(JB) 
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The Honorable Dale E. Kildee 
Chairman, Subcommittee on Elementary, 

Secondary, and Vocational Education 
Committee on Education and Labor 
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Dear Mr. Chairman: 

In response to your request, this report presents the results of our review of implementation of 
the Chapter 1 program improvement provisions. 

Copies of this report are also being sent to appropriate House and Senate Committees, the 
Secretary of Education, and other interested parties. 

This report was prepared under the direction of Linda G. Morra, Director, Education and 
Employment Issues, who may be reached on 512-7014 if you or your staff have any questions 
about it. Other m^jor contributors are listed in appendix XII. 

Sincerely yours, 




Lawrence H. Thompson 
Assistant Comptroller General 
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Executive Summary 



Chapter 1 is the largest federal education program for children in 
elementary and secondary schools. With $6. 1 billion in federal funds in 
fiscal year 1992, Chapter 1 serves over 5 million children, through 
supplemental instruction in reading, math, or language arts, in about 
51,000 schools. The statutory goals of Chapter 1 are to help educationally 
deprived children 1 (1) succeed in the regular program of the school district, 
(2) attain grade-level proficiency, and (3) improve their achievement in 
basic and more advanced skills. To ensure that individual schools are 
effective in helping Chapter 1 students achieve these goals, the Congress 
created a new accountability system, focused on student outcomes. This 
accountability system was established by the program improvement 
provisions of the Hawkins-Stafford Elementary and Secondary School 
Improvement Amendments of 1988 (P.L. 100-297). These provisions reflect 
a national movement toward holding schools accountable for student 
outcomes. 



The program improvement provisions require local education agencies 
(hereafter "school districts") to identify schools with ineffective Chapter 1 
programs and work with these schools to develop and implement local 
program improvement plans. If a school's Chapter 1 program does not 
show sufficient improvement under the local plan, the state education 
agency must become involved, working with local officials to develop and 
implement a joint improvement plan for the school. The Congress 
intended program improvement plans to incorporate program changes 
most likely to improve student performance. 

Local and state education officials and recent studies have raised 
questions about the accuracy of the process used to identify schools for 
program improvement. Recent studies have also raised concerns about 
schools' local program improvement efforts, although no studies have 
focused on state and local efforts in the joint phase. 

In anticipation of the 1993 reauthorization of Chapter I, the Chairman, 
Subcommittee on Elementary, Secondary, and Vocational Education, 
House Committee on Education and Labor, asked gao to (1) assess the 
process used to identify schools in need of program improvement and 
determine whether this process could be improved and (2) compare 
implementation of the joint and local phases of program improvement, 
including the roles of school, district, and state staff, as well as the 
program changes (or strategies) schools used to bring about improvement. 



*The Department of Education defines educationally deprived children as children whose educational 
attainment is below the level that is appropriate for their age. 
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rfr tfrm in H ^ school must be identified for program improvement if its Chapter 1 

r><±Civ.gl U III lu students fail to make substantial progress toward meeting specific 

"desired outcomes," which represent the school district's educational 
goals for Chapter 1 children. At a minimum, districts must establish 
desired outcomes concerning average student gains in basic and advanced 
skills, as measured by achievement tests. 2 States must specify, in their state 
program improvement plans, the minimum standard for average test score 
gains from one year to the next. Under the federal minimum standard, 
which states may exceed, schools are identified if students' average test 
scores stay the same or decrease. 

To provide a more complete picture of program effectiveness, however, 
the Department of Education encourages districts to establish additional 
desired outcomes, measured by other indicators of student performance. 
Districts could, for example, establish a desired outcome concerning 
Chapter 1 students' mastery of certain skills, measured by their 
performance on criterion-referenced tests, 3 or a desired outcome 
concerning success in the regular program, measured by the length of time 
students remain in Chapter 1. 

The joint phase of program improvement focuses state attention on 
schools whose Chapter 1 programs did not improve sufficiently during the 
local improvement phase. During the joint phase, in contrast to the local 
phase, state agencies must become directly involved with local officials in 
developing and implementing program improvement plans for identified 
schools. About 1,400 schools (about 3 percent of all Chapter 1 schools) 
entered the joint phase in school year 1991-92, the first year large numbers 
of schools entered this phase of program improvement. 

To assess the process for identifying schools with ineffective Chapter 1 
programs, gao analyzed a data set containing achievement-test scores for 
Chapter 1 students in one large state and reviewed research literature on 
the effects of achievement testing on classroom instruction. To obtain 
information about implementation of the joint phase of program 
improvement in comparison with the local phase, gao (1) surveyed the 
Chapter 1 coordinators of all 50 states and the District of Columbia, as 
well as officials in nearly every district and school involved in the joint 



2 Whcn wc refer U> "achievement tests," we mean standardized, norm-referenced, multiple-choice 
achievement tests such as the Metropolitan Achievement Test, Comprehensive Test, of Basic Skills, or 
lowa Test of Basic Skills. 

•^Criterion-referenced tests, often developed by state education agencies or school dist ricts, measure 
student mastery of state or locally defined educational objectives. 
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phase during school year 1991-92, and (2) conducted case studies in four 
states, interviewing state, district, and school officials about their 
experiences with both phases of program improvement. 



PpQiiltQ in Rripf ^ e P rocess use d to identify schools for program improvement is flawed 

because schools often are evaluated only on whether their Chapter 1 
students show annual increases in achievement-test scores, gao's analysis 
shows that when schools are evaluated only on achievement test scores, 
many schools are likely to be judged as effective or ineffective on the basis 
of changes in test scores that reflect random fluctuations, rather than 
actual changes in student performance. Achievement-test scores alone 
cannot provide a complete picture of program effectiveness. However, the 
statute, regulations, and Chapter 1 Policy Manual do not adequately 
explain how evidence from multiple indicators of student performance 
may be considered when identifying schools for program improvement. 

In addition, identifying schools on annual changes in achievement-test 
scores may neglect the improvement needs of schools with students that 
make annual gains on test scores, but remain far below grade level. Thus, 
the current identification process may not hold schools accountable for 
the Chapter 1 goal of helping students attain grade-level proficiency. 
Finally, when increasing achievement-test scores is seen as the most 
important objective for Chapter 1, schools may try to raise students' test 
scores without improving instructional practices, using less desirable 
strategies that narrow the scope of instruction to focus on material 
covered by the tests. These efforts may be inconsistent with the Chapter 1 
goal of helping children succeed in the regular program of the district. 

Holding schools accountable for achieving multiple desired outcomes and 
assessing program effectiveness with multiple indicators of student 
performance would improve the identification process. This would also 
help reduce the emphasis schools place on achievement-test scores in 
providing instruction to their Chapter 1 students. 

School staffs were considered more influential than district and state staff 
in developing improvement plans in both the local and joint phases of 
program improvement. In both phases, state assistance was typically more 
general than specific — focusing more on explaining the requirements of 
program improvement than on helping individual schools develop 
improvement plans — although specific state assistance did increase in the 
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joint phase. The amount oi specific assistance schools received was 
related to the number of state staff available to work with them. 

Schools continued to use most of the same improvement strategies in the 
joint phase of program improvement that they used in the local phase. The 
most widely used strategies in both phases were (1) improving 
coordination between Chapter 1 and the regular instructional program and 
(2) increasing parental involvement. In addition, large-city schools were 
more likely than schools in other locations to adopt the strategies of 
adding a summer program or adding an extended-day program for 
Chapter 1. 



Principal Findings 



Program Effectiveness 
Often Judged Only on 
Achievement-Test Scores 



States and school districts often identify schools for program 
improvement solely or unconditionally on the basis of students 1 
achievement-test scores. Many school districts have only established 
desired outcomes for their Chapter 1 programs concerning students 1 
average gains on achievement tests, thus judging program effectiveness 
with a single indicator of student performance. A majority of school 
districts have established additional desired outcomes for their Chapter 1 
program, but most of these districts still use test scores as an 
unconditional criterion when identifying schools: If a school's students do 
not show sufficient gains on achievement tests, the school is identified for 
program improvement, regardless of student success on other desired 
outcomes. (See pp. 23-24.) 

However, the law suggests that the identification process can be more 
flexible. If a school's Chapter 1 students do not make sufficient gains on 
achievement tests, a school, it can be argued, need not be identified for 
program improvement — so long as evidence from other indicators shows 
that the Chapter 1 program is effective. The practice oi using gains on 
achievement tests as an unconditional criterion in identifying schools may 
stem from the language of the statute and a lack of guidance in the 
Department's regulations and Chapter 1 Policy Manual. (See pp. 24-25.) 



Reliance on Achievement 
Tests Reduces Accuracy of 
Identification Process 



When Chapter 1 program effectiveness is judged only on changes in 
achievement-test scores, many schools are likely to be identified or not 
identified for program improvement on the basis of random fluctuations in 
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test scores, rather than actual changes in student performance. For 
25 percent of the schools in gao's analysis of one state's schools, a 
confident judgement could not be made about whether their students 
actually exceeded or fell short of the standard for gains on achievement 
tests. These results are based on the federal minimum standard for gains; 
by using a standard that required greater gains, many more schools would 
have average changes in test scores that could not be distinguished from 
random fluctuations. (See pp. 25-26.) 



Focus on Annual Gains in 
Test Scores May Neglect 
Needs of Schools With 
Lowest Achieving Students 



Focusing on annual gains in achievement-test scores may direct the 
attention of local and state officials away from schools whose students 
show gains in test scores but nevertheless remain far below grade level. 
Thus, the current identification process may not hold schools accountable 
for the Chapter 1 goal of helping students attain grade-level proficiency. 
gao found that schools whose students initially nad the lowest average 
achievement-test scores were less likely to be identified for program 
improvement than schools whose students initially had the highest 
average test scores. For example, under the federal minimum standard for 
test score gains, about 10 percent of schools with the lowest initial average 
test scores would be identified as needing improvement, compared with 
about 14 percent of schools with the highest initial average test scores. 
Using a higher standard for test score gains, the difference in the 
percentage of schools identified in these two groups increases from 
4 percent to 13 percent. (See pp. 28-29.) 



Pressure to Increase Test 
Scores May Have Negative 
Effect on Chapter 1 
Instruction 



Pressure to increase achievement-test scores may have a negative effect 
on the instruction Chapter 1 students receive. Research evidence indicates 
that when such pressure exists, subject areas covered on achievement 
tests tend to be taught to the exclusion of untested subject areas, and 
instruction tends to be oriented toward promoting students' ability to 
recognize correct answers to multiple-choice questions rather than 
promoting higher order thinking skills. In its case studies, gao found that 
some schools identified for the joint phase of program improvement 
simply re-targeted existing instructional practices on the subject areas in 
which test scores were too low, rather than scrutinizing their instructional 
program and adopting new approaches. Focusing instruction on increasing 
students* achievement-test scores may not help Chapter 1 students 
succeed in the regular instructional program of the school district — a 
statutory goal of Chapter 1. The Department of Education cautions, in its 
Chapter 1 Policy Manual, that gains on achievement tests may not 
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translate into improved performance in regular classrooms. (See 
pp. 29-30.) 



School Staffs Had Most 
Influence on Improvement 
Efforts During Both Phases 



gao's survey of district Chapter 1 coordinators found that in both the joint 
and local phases, school staffs had more influence than districts and states 
in determining schools' improvement needs and selecting improvement 
strategies. State staffs were the least influential among these three groups. 
Almost 80 percent of district coordinators rated school staffs as having a 
"very great" or "great" influence, during the joint phase, in determining 
improvement needs and selecting strategies. About 70 percent of the 
coordinators felt their own staff had this degree of influence, and about 
40 percent said state staffs did. Coordinators in districts that received 
greater amounts of state assistance rated state influence higher than 
coordinators in districts that received less assistance. (See pp. 34-36.) 



Improvement Strategies 
for Joint Phase Similar 
to Local Phase 



Most schools in the joint phase continued to use the improvement 
strategies they began using in the local phase. About two-thirds of 
principals reported that the improvement strategies their schools were 
using in the joint phase were "exactly the same" or "very similar" to those 
adopted in the local phase. The two most common strategies in both 
phases were (1) improving coordination between Chapter 1 and the 
regular instructional program and (2) increasing parental involvement. 
During both phases, each of these strategies was used in about 90 percent 
of the schools. In addition, over 75 percent of the principals gao surveyed 
said that instructing students on test-taking skills was one of their 
strategies during each phase. Large-city schools adopted some strategies 
much more often than schools in other locations during both phases. In 
the joint phase, about 62 percent of large-city schools offered summer 
programs for Chapter 1 (compared with about 22 percent of all other 
schools) and about 53 percent provided Chapter 1 services through 
extended-day programs (compared with about 19 percent of all other 
schools). (See pp. 37-43.) 



State Assistance Increased 
During the Joint Phase 



State education agencies provided more technical assistance to districts 
and schools in the joint phase than in the local phase. But in both phases, 
state agencies provided greater amounts of general assistance, such as 
explaining requirements, than specific assistance, such as improving 
individual schools. Specific state assistance, however, increased in the 
joint phase. About 31 percent of principals reported a "very great" or 
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"great" amount of specific state assistance in the joint phase, up from 
23 percent in the local phase. Among district coordinators, the percentage 
reporting this amount of specific assistance increased from 30 percent in 
the local phase to 46 percent in the joint phase. In both phases, however, 
some schools and districts reported receiving little state assistance. In the 
joint phase, for example, about 20 percent of district coordinators and 
37 percent of principals reported a "very small" amount or "no" specific 
assistance from their states. Principals were most likely to report a "very 
small" amount or "no" specific state assistance if they were from states in 
which small numbers of state staff were responsible for assisting large 
numbers of joint-phase schools. (See pp. 43-48.) 

Because this study examined the first school year in which large numbers 
of schools entered the joint phase, gao's findings may not reflect how the 
joint phase will be incremented in subsequent years. For example, as 
district and school ofucials become more familiar with the requirements of 
the joint phase, state officials may be able to spend less time providing 
general information and more time assisting individual schools with their 
improvement efforts. 



To improve the process used to identify schools for program improvement 
and to help reduce the emphasis placed on standardized achievement-test 
results, gao recommends that the Congress amend the Elementary and 
Secondary Education Act to require (1) states to establish, for schools' 
Chapter 1 programs, multiple desired outcomes related to the statutory 
goals of Chapter 1 and (2) districts to assess program effectiveness by 
considering whether evidence from multiple indicators of student 
performance shows substantial progress in achieving these outcomes. 

The Congress should require state education agencies to specify, in their 
state program improvement plans, (1) the desired outcomes for Chapter 1 
schools, (2) the indicators that will be used to measure student progress 
toward those desired outcomes, (3) minimum standards for student 
performance on each indicator, and (4) a definition of substantial progress 
toward meeting the desired outcomes as a group (that is, how districts will 
weigh evidence from multiple indicators in judging whether their 
Chapter 1 schools are effective). Districts should also be allowed to set 
higher standards than required by their state education agency and to use, 
with the approval of their state agency, additional or alternative desired 
outcomes and indicators. 



Recommendations 
to the Congress 
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gao also recommends clarifications to statutory language on the process 
used to identify schools for program improvement (see p. 32). 



Matter for 

Congressional 

Consideration 



To help ensure that states establish adequate standards for identifying 
Chapter 1 schools in need of improvement, the Congress should consider 
amending the Elementary and Secondary Education Act to require that the 
Secretary of Education review and approve the desired outcomes and 
indicators specified by states in their state program improvement plans. 
This review could focus on determining whether states have specified 
(1) desired outcomes and indicators that reflect high educational 
standards and pertain to the statutory goals of Chapter 1 and (2) a 
reasonable definition of substantial progress toward meeting multiple 
desired outcomes. 



The Department of Education agreed with gao's conclusions, but 
Agency Comments cautioned that multiple indicators (1) can sometimes provide conflicting 

evidence about program effectiveness and (2) need to be valid and reliable 
measures (see app. VIII). gao believes that the recommendations and 
matter for congressional consideration in this report address these 
concerns. Situations involving conflicting evidence should be resolved 
according to state plans, which will specify how evidence from all 
indicators will be considered together in evaluating Chapter 1 program 
effectiveness. In addition, if the Congress requires the Department to 
review and approve each state's program improvement plan, the 
Department could play a key role in ensuring that proposed indicators are 
sufficiently valid and reliable. 
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Introduction 



Chapter I is the largest federal program for elementary and secondaiy 
education, with a fiscal year 1992 budget of $6.1 billion. These funds 
support supplemental instruction in reading, math, or language arts for 
over 5 million children in about 51,000 schools, including about 70 percent 
of the nation's elementary schools. The statutory goals of Chapter 1 are to 
help educationally deprived children 1 (1) succeed in the regular program of 
the school district, (2) attain grade-level proficiency, and (3) improve their 
achievement in basic and more advanced skills. 

During the 1988 reauthorization of Chapter 1, the Congress created a new 
accountability system, focused on student outcomes, to ensure that 
individual schools are effective in helping students achieve the goals of 
Chapter 1. This accountability system is outlined in the program 
improvement provisions of the Hawkins-Stafford Elementary and 
Secondaiy School Improvement Amendments of 1988 (P.L. 100-297). 2 
Under these provisions, local education officials are required to identify 
schools with ineffective Chapter 1 programs and then develop and 
implement plans to improve those programs. If thes^ local efforts do not 
result in sufficient improvement, state officials must become involved with 
local officials in the program improvement process. 

Some local and state education officials, as well as other experts, have 
questioned the accuracy of the process used to identify schools for 
program improvement. 3 They are concerned that some of the schools most 
in need of improvement may not be identified and that some schools are 
identified even though their Chapter 1 programs are effective. In addition, 
recent studies of program improvement have raised concerns about the 
program improvement efforts undertaken by identified schools. 4 These 
studies have focused mainly on the local phase of program improvement; 
none have specifically examined joint state-local efforts. 



'The Department of Education defines educationally deprived children as children whose educational 
attainment is below the level that is appropriate for their age. 

^he law previously required that districts use evaluation results to improve their Chapter 1 programs; 
however, until the Hawkins-Stafford Amendments, no system existed to hold schools accountable for 
doing so. 

3 For views about the accuracy of the identification process among the officials we surveyed, see 
appendix I. 

' National Assessment of the Chapter 1 Program: The Interim Report , U.S. Department of Education 
(Washington, D.C.: June 1992); Mary Ann Millsap and others, The Chapter 1 Implementation Study: 
Interim Rep ort (Cambridge, Mass.: Abt Associates, Inc., under contract with the U.S. Department of 
Education, Gffice of Policy and Planing, 1992). 
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Chapter 1 
Introduction 



Evaluating Chapter 1 
Program < L 
Effectiveness 



The program improvement provisions require local education agencies 
(hereafter "school districts") to determine annually whether their schools' 
Chapter 1 programs have been effective i improving the performance of 
participating students. The statute indicates that a school shall be 
identified for program improvement if its Chapter 1 students (1) show no 
improvement or a decline in "aggregate performance" or (2) fail to make 
substantial progress toward meeting the school's "desired outcomes." 
Aggregate performance refers to the average change in students' scores on 
achievement tests from one year to the next; 5 desired outcomes are the 
educational goals, in terms of basic and more advanced skills, that school 
districts set for participating children. Chapter 1 regulations, however, 
permit districts to use average gains on achievement tests (in basic and 
advanced skills) as the only desired outcomes for their schools' Chapter 1 
students. 



The Department of Education encourages districts to establish additional 
desired outcomes, measured by other indicators of student performance, 
because this will provide a more complete picture of program 
effectiveness. 6 Districts could, for example, establish a desired outcome 
concerning Chapter 1 students' mastery of certain skills, measured by 
students' performance on criterion-referenced tests, 7 or a desired outcome 
concerning success in the regular program, measured by the length of time 
students remain in Chapter 1. 



Annual Achievement 
Testing in Chapter 1 
Schools 



By judging the effectiveness of individual schools' Chapter 1 programs on 
the basis of students' achievement-test scores, the program improvement 
provisions significantly raised the stakes associated with achievement 
tests in Chapter 1 schools. Previously, schools were not held accountable 
for increasing students' scores on achievement tests; now, however, 
schools are identified as ineffective if their students do not show sufficient 
achievement gains from one year to the next. 

Once a year, all of a school's Chapter 1 students in grades 2 to 12 must 
take an achievement test to measure performance in the subject areas in 



5 Whcn we refer to "achievement tests," we mean standardized, norm-referenced, multiple-choice 
achievement tests such as the Metropolitan Achievement Test, Comprehensive Test of Basic Skills, or 
Iowa Test of Basic Skills. 

fl Chapter 1 Policy Manual: Basic Programs Operated by Local Educational Agencies , U.S. Department 
of Education, Office of Elementary and Secondary Education, Compensatory Education Programs 
(Washington, D.C.: Apr. 1990), pp. 120-21 and 166. 

7 Criterion-rcferenccd tests, often developed by state education agencies or school districts, measure 
st udent mastery of state or locally defined educational objectives. 
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which these students receive supplemental instruction. Each student's 
score is reported in terms of normal curve equivalents (nces), a special 
scale used for achievement testing in Chapter I. 8 Under the program 
improvement provisions, schools are evaluated on the average change in 
their Chapter 1 students* nce scores over a Lyear period. To meet this 
requirement, an "nce change score" is calculated for each Chapter 1 
school, using only the scores of students who took an achievement test in 
2 successive years (referred to as "matched test scores"). 9 For example, if a 
school's average nce score is 32 nces one year and 34 nces the next year, 
then its nce change score is 2 nces. When a school's nce change score is 
positive, the Chapter 1 students are said to have made gains in 
achievement; when it is negative, the students are said to have shown 
losses in achievement; and when it is 0, the students are said to have 
maintained the same achievement level. 

The law requires state education agencies to specify, in their state program 
improvement plans, the standard their districts will use in evaluating 
whether a school's Chapter 1 students have made sufficient achievement 
gains. The federal minimum standard, established by regulation, requires 
that to make sufficient nce gains, a school's nce change score must be 
greater than 0. We refer to this as the u 0 nce standard." However, states 
and districts may establish higher nce standards, such as requiring schools 
to make gains greater than 2 or 4 nces. Most states and districts use a 0 nce 
standard, although the number using an nce standard higher than the 
federal minimum has increased since the first year the Chapter 1 
accountability system took effect. 



NCE Change Scores 
Provide an Imprecise 
Measure of Student 
Achievement 



nce change scores are not precise measures of average changes in student 
performance on achievement tests. Students' actual gains or losses may be 
somewhat higher or lower than the nce change score indicates. The 
precision of this estimate can be affected by a variety of factors, including, 
for example, the reliability of the achievement test used and the number of 
students with matched test scores from which a school's nce change score 
is calculated. 
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*An NCE score, like a percentile rank, shows how well a student performed on an achievement test 
Unlike percentiles, however, NCE scores for different students can be averaged. This allows NCEs to 
be aggregated to provide a national picture of Chapter I program effectiveness, which is why NCEs 
were originally developed. 

*The use of matched test scores ensures that a school's NCE change score reflects only the 
performance of students who were in the Chapter 1 program for a full year, not those who attended 
the school just part of the school year. 
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The concept of test reliability refers to how similar a student's score would 
be if he or she took the same test many times. Although the achievement 
tests in wide use today have a high degree of reliability, no achievement 
test is perfectly reliable; some amount of random fluctuation in a student's 
scores would be normal and expected. The number of students 
contributing to a school's nce change score can also affect its precision as 
an indicator of average student achievement. An nce change score 
calculated from a small number of matched test scores is more likely to be 
affected by random fluctuations in students' test scores than one 
calculated from a large number of matched scores. 

By accounting for sources of imprecisic n, however, it is possible to 
determine, with a high degree of confidence, the range within which a 
school's actual nce change score falls. For example, this range might 
extend 2 nces above and below the school's nce change score. If this range 
does not extend across the nce standard (as with schools A and B in 
fig. 1.1), there is very little doubt as to whether the school's students 
exceeded the standard for nce gains (school A) or fell short of it (school 
B). However, if this range does extend across the nce standard (as with 
schools C and D in fig. 1.1), a confident judgement cannot be made about 
whether the school's students actually exceeded or fell short of the nce 
standard. We say these schools have "inconclusive nce change scores" 
because they were not above or below the nce standard by a statistically 
significant margin. 
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Figure 1.1: Some Schools' NCE 
Change Scores May Not Permit a 
Confident Judgement About Whether 
Students Exceeded or Fell Short 
of the NCE Standard 
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Developing and 
Implementing Local 
and Joint Program 
Improvement Plans 



When a school is initially identified as ineffective, the school district, in 
coordination with the school, must develop and implement a local plan for 
program improvement The regulations allow a maximum of 1 full school 
year to develop these plans, but require that portions be implemented as 
soon as possible. In addition, the statute specifies that program 
improvement plans should incorporate the program changes that have the 
greatest likelihood of improving student performance. Examples of 
program changes (hereafter, "improvement strategies") include adopting a 
new instructional approach or offering Chapter 1 services during the 
summer. 



If a school*s Chapter 1 program is again identified as ineffective, after the 
local plan has been fully implemented for 1 full school year, the state 
education agency and the district together must develop and implement a 
joint program improvement plan for the school, in coordination with 
school staff. If the joint plan is unsuccessful, the state and district must 
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review and revise the plan each year until Chapter 1 students show 
improved performance over more than 12 months. 

About 5,000 schools entered the local phase of program improvement 
during school year 1989-90, the first year the new accountability system 
took effect. In school year 1991-92, about 1,400 schools entered the joint 
phase; most of these schools had not shown sufficient improvement 
during 2 years in the local phase. 10 In most states, school year 1991-92 was 
the first year that schools entered the joint phase. 



In anticipation of the 1993 reauthorization of Chapter 1, the Chairman, 
Subcommittee on Elementary, Secondary, and Vocational Education, 
House Committee on Education and Labor, asked gao to conduct a study 
of the Chapter 1 program improvement process, focusing on the accuracy 
of the school identification process and implementation of the joint phase. 

In response to the request, we agreed to (1) assess the process used to 
identify schools for program improvement and determine whether it could 
be improved and (2) compare implementation of the joint and local phases 
of program improvement, including the roles of school, district, and state 
staff, as well as the strategies schools used to bring about improvement. 
To meet these objectives, we used multiple methodologies, including 
statistical analyses, mail surveys, case studies, and a literature review. 



Statistical Analyses To demonstrate the extent of potential inaccuracy in the process used to 

identify schools for program improvement, Wc analyzed a data set 
containing achievement-test scores for Chapter 1 students in 
Pennsylvania. We limited our analysis to students* test scores on reading 
comprehension subtests, which are used to measure advanced reading 
skills. We also limited our analysis to the 2,1 15 schools that had more than 
10 students with matched test scores. 11 

We first determined the number of schools with nce change scores that did 
not permit a confident judgement about whether their students actually 
exceeded or fell short of a given standard for nce gains; that is, the number 
of schools with inconclusive nce change scores. We also determined the 



l0 For information on the number and characteristics of schools and districts involved in the joint phase 
during school year 1991-92, see appendix II. 

u Our rationale for those limits on the scope of our statistical analyses, as well as the technical details 
of these analyses, are presented in appendix III. 



Objectives, Scope, 
and Methodology 
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extent to which (1) schools with matched test scores for small numbers of 
Chapter 1 students are more likely to have inconclusive nce change scores 
than schools with matched test scores for large numbers of students and 
(2) schools with initially high average nce scores are more likely to be 
identified than schools with initially low average nce scores. 



Mail Surveys To obtain information about implementation of the joint phase of program 

improvement in comparison with the local phase, we conducted three mail 
surveys. First, in December 1991, we surveyed the state Chapter 1 
coordinators of all 50 states and the District of Columbia. Our response 
rate was 100 percent. The questionnaire asked state coordinators about 
their policies on program improvement and their opinions on the accuracy 
of the identification process; in addition, the survey asked them !:o send us 
mailing lists of every school and district identified for the joint phase of 
program improvement during school year 1991-92. 

We received usable lists from 46 states, identifying a total of 1,397 schools 
and 498 districts involved in the joint phase. Iowa, North Dakota, and the 
District of Columbia did not submit any mailing lists because they had no 
schools in the joint phase during school year 1991-92. Florida did not 
provide us with a list of its joint-phase schools and districts in time for us 
to include them in our surveys of principals and district coordinators. 
Finally, although California submitted a list of schools and districts, we did 
not include them in our surveys because the state education agency makes 
no distinction between the local and joint phases of program 
improvement. 12 

Then, in April 1992, we surveyed (1) the Chapter 1 coordinators of each of 
the 498 districts and (2) the principals of all 1,397 schools. Our response 
rates were 96 percent for district coordinators and 86 percent for school 
principals. 13 Among other things, the questionnaires for these surveys 
asked about the roles of school, district, and state staff in determining 



l2 California has implemented the Chapter 1 program improvement provisions differently from all other 
states. California considers all program improvement activities to be a joint state-local effort 
Furthermore, all California schools identified for program improvement must stay in for 4 school 
years — 1 year to develop a plan, followed by 3 years of implementation. Even if a California school 
shows success after I or 2 years, it must stay in program improvement for the remainder of the 4-year 
period. 

13 Although we surveyed the vast majority of schools and districts involved in the joint phase, lists 
provided by states were in some cases incomplete and in other cases may have had some errors. For 
example, some respondents reported that their districts or schools were not involved in the joint 
phase; this may indicate some confusion between state and local officials about which schools were 
actually in the joint phase (see app. II). 
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program improvement needs and selecting improvement strategies; the 
strategies schools used in each phase; and the accuracy of the 
identification process. We also asked the principals how far along their 
schools were in developing and implementing their joint plans; over 
70 percent of the principals indicated they had partially or fully 
implemented their joint plan. (State, district, and school questionnaires are 
presented in apps. IV, V, and VI, respectively.) 



To obtain more information about implementation of the joint phase in 
comparison with the local phase, we conducted case studies in four states: 
Arkansas, Maryland, Michigan, and Mississippi. We chose these states 
primarily because they each had a large number of schools in the joint 
phase during school year 1991-92. Within each state, we interviewed the 
state Chapter 1 coordinator and other state staff knowledgeable about 
program improvement activities. We also visited one urban school district 
and one rural district in each state; in both districts, we interviewed the 
district Chapter 1 coordinators and any key members of their staffs. Within 
each district we visited one school, where we interviewed the school 
principal and one or more Chapter 1 instructors. 

We visited only elementary schools because most Chapter 1 services in the 
nation are provided at this level. We also chose schools with a high 
concentration of poverty because we believe such schools face the 
greatest challenge in improving the achievement of their Chapter 1 
students. 14 Other factors we considered in choosing districts and schools 
included (1) the length of time the district coordinator and principal had 
held their current positions (because we wanted to interview officials with 
first-hand knowledge of activities in both phases of program 
improvement) and (2) the degree to which joint improvement plans had 
been implemented (because we wanted to visit places where the joint 
phase was significantly under way). (The districts and schools we visited 
in each state are listed in app. VII.) 

At the time of our study, several other studies of program improvement 
had been completed or were under way, some as part of the Department of 



14 We measured poverty concentration as the percentage of students participating in the free or 
reduced-price lunch program. Children who receive a free lunch come from families with incomes of 
130 percent or less of the official poverty threshold, which was $10,860 for a three-person family in 
1991; children who receive a reduced-price lunch come from families with incomes of 185 percent or 
less of the official poverty threshold. (Sources: U.S. Department of Commerce, Bureau of the Census, 
and U.S. Department of Agriculture, Child Nutrition Division.) 
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Education's national assessment of the Chapter 1 program. We reviewed 
the findings from these studies that are related to program improvement 
and refer to them in this report, when applicable. We also reviewed 
literature on the effects of standardized testing on school instruction. 
References for the literature we reviewed appear in the bibliography. 



The Department of Education provided written comments on a draft of 
this report. These comments are presented in appendix VTIL We revised 
our report, on the basis of these comments, when applicable. 

We carried out our study between July 1991 and November 1992 in 
accordance with generally accepted government auditing standards. 
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The accountability system established by the Chapter 1 program 
improvement provisions may not accurately identify schools with 
ineffective Chapter 1 programs nor encourage schools to make program 
changes that are most likely to help students achieve the goals of 
Chapter 1. These problems stem from evaluating schools primarily on 
students' achievement-test scores. In addition, because schools are judged 
effective when they show annual gains in test scores, states and districts 
may neglect the improvement needs of schools whose students make gains 
but still remain far below grade-level. 

The use of additional indicators of student performance would provide a 
more complete picture of Chapter 1 program effectivt ness. This would 
also reduce the emphasis schools place on achievement tests in selecting 
improvement strategies. But the statute, regulations, and Chapter 1 policy 
guidance inadequately explain now evidence from multiple indicators may 
be considered when identifying schools in need of program improvement 



Chapter 1 

Effectiveness Often 
Judged Only on 
Achievement-Test 
Scores 



Many states and districts judge the effectiveness of their schools' 
Chapter 1 programs solely or unconditionally on the basis of students' 
average achievement-test scores. Many school districts have only 
established desired outcomes concerning Chapter 1 students' average 
gains in basic and advanced skills, as measured by their performance on 
achievement tests. A majority of districts, however, have established 
additional desired outcomes, for example, concerning Chapter 1 students' 
mastery of particular skills, as measured by their scores on 
criterion-referenced tests. 1 When judging whethei a school's Chapter 1 
program needs improvement, however, few of these districts consider 
student progress toward these additional desired outcomes together with 
achievement-test scores. Rather, achievement-test scores are used as an 
unconditional criterion: If a school's Chapter 1 students do not show 
sufficient nce gains, the school is identified as needing program 
improvement, regardless of students* success in meeting the other desirsd 
outcomes. 2 
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l These additional desired outcomes often pert* - o achievement-test results, but districts use some 
measure other than the average gains among all participating students over a 1-year period. For more 
information on the desired outcomes (other than NCE gains) used in the districts we surveyed, see 
appendix IX. 

2 Among the district coordinators we surveyed whose district required the use of desired outcomes 
other than average NCE gains, 80 percent indicated that a school that does not show sufficient NCE 
gains must be identified for program improvement, even if it has other evidence of the effectiveness of 
its Chapter 1 program. 

p a g C 23 GA0/HRD-9.1-69 Chapter I AccounUbllity 



Chapter 2 

Chapter 1 Accountability System Hindered 
by Reliance on Achievement Tests 



School districts may use nce change scores as the sole criterion in 
evaluating Chapter 1 program effectiveness for a variety of reasons: First, 
for Chapter 1 students in grades 2 to 12, districts are not required to set 
any desired outcomes other than annual gains on achievement tests. 
Second, districts may have little experience in setting outcome-based goals 
for their Chapter 1 programs and in measuring students' progress toward 
meeting those goals. 3 A primary reason why many districts use nce change 
scores as an unconditional criterion may be the commonly held view that 
the law requires this. One provision of the statute indicates that a school 
must be identified for program improvement if its Chapter 1 students 
either (1) do not make sufficient nce gains or (2) fail to make substantial 
progress toward meeting other desired outcomes. 4 

However, another provision in the same section of the law suggests that 
the identification process can be more flexible, specifically, that a school 
does not have to be identified if its Chapter 1 students do not make 
sufficient nce gains. This provision specifies five "local conditions" that 
states and districts must take into consideration when identifying schools. 
One of these local conditions, it can be argued, permits states and districts 
to disregard achievement-test scores if other evidence demonstrates the 
positive effects of Chapter 1 on participating students. 5 

Many local and state officials are not using this flexibility, perhaps because 
it is not explained in the Department of Education's regulations or in its 
Chapter i Policy Manual. Both the Chapter 1 regulations and Policy 
Manual indicate that a school must be identified if it does not show 
sufficient nce gains. The Policy Manual suggests that districts and states 
exempt schools from program improvement only on the basis of local 



See, for example, Millsap and others, The Chapter 1 Implementation Study and Chapter 1 Program 
Improvement and Innovation Across the Stages: An Overview and Slate Profiles, Council of Chief Suite 
School Officers (CCSSO), (Washington, D.C.: 1992). 

'Public Law 100-297 section 1021(b). This wording in the law may be another reason why some states 
and districts use NCE change scores as the sole criterion in evaluating Chapter 1 program 
effectiveness. One recent study said that "many localities see the dual set of standards as a form of 
'double jeopardy/" in which additional desired outcomes only increase the chances that a school will 
be identified. (Source: Chapter 1 Program Improvement and Innovation Across the States: An 
Overview and State Profiles , CCSSO, p. 5). This study, as well as others, concluded that this section of 
the statute has reduced the incentive to set additional desired outcomes. Sec, for example, Sam 
Stringfield and others, "Chapter 1 Program Improvement Cause for Cautious Optimism and a Call for 
Much More Research," Educational Evaluation and Policy Analysis , Vol. 13, No. 4 (Winter 1991), 
pp. 399-406; Nancy Kober, "The Role and Impact of Chapter 1 , ESEA, Evaluation and Assessment 
Practices," prepared for the U.S. Congress, Office of Technology Assessment, June 1991. 

f, Public Law 100-297 se n 1021(e). The local conditions included in this section of the statute are 
listed in appendix X 
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conditions that are unforeseen, such as an unexpected increase in student 
mobility. 

As a matter of policy, Department officials support the local condition 
permitting districts to disregard achievement-test scores when other 
evidence indicates that a school's Chapter 1 program is effective. They 
also believe, however, that some of the other local conditions specified in 
the statute are not appropriate reasons for exempting schools from 
program improvement One local condition, for example, would allow 
state and local officials to exempt a school if they felt the extent of 
educational deprivation among its Chapter 1 students had a negative 
impact on improvement efforts. Department officials believe, and we 
concur, that this would be contrary to the Chapter 1 aim of serving 
educationally deprived children. Department officials explained that they 
find it difficult encouraging state and local officials to consider some local 
conditions when identifying schools while at the same time discouraging 
these officials from considering others. 



Reliance on 
Achievement-Test 
Scores Reduces 
Accuracy of 
Identification Process 



Sole reliance on achievement-test scores reduces the accuracy of the 
process used to identify schools for program improvement. States and 
districts are likely to judge the effectiveness of many schools 1 Chapter 1 
programs on the basis of achievement-test scores that reflect random 
fluctuations rather than actual changes in student achievement. Using the 
federal minimum standard for nce gains (0 nce standard), we found that 
about 25 percent of the 2, 1 15 schools in our analysis would be judged as 
effective or ineffective on the basis of inconclusive nce change scores; that 
is, a confident judgement cannot be made as to whether schools' actual 
scores were above or below the standard. An even greater percentage of 
schools would be judged on the basis of inconclusive scores using higher 
nce standards (see fig. 2.1). 6 



*This is because as the minimum standard is raised toward the average NCE change c^re, more 
schools will have scores closer to t he standard. The closer a school's score is Us the standard, the 
greater the chance that the range within which its actual score falls will extend across that standard. 
The average NCE change score for the schools included in our analysis was 5.1 NCEs for advanced 
skills in reading— eoasiderably higher than the federal minimum standard of 0 NCEs. 
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Figure 2.1 : Percentage of Schools With ■■M^^^HH 
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Greatest Potential for 
Inaccuracy Among Schools 
With Test Scores for Small 
Numbers of Students 



Schools with matched test scores for small numbers of students are most 
likely to be inaccurately judged for program improvement. This is because 
the range within which a school's actual nce change score might fall is 
greater for a school with a small number of matched scores than it would 
be for a school with a large number of matched scores. 7 The wider this 
range, the more likely that it would extend across the nce standard, 
meaning that the school would have an inconclusive change score. 
Schools with a small number of matched scores may include (1) rural 
schools, which tend to serve small numbers of students, and (2) urban 
schools, which tend to serve large numbers of students but, often, not the 
same students over 2 successive school years because of high mobility 
rates. 



Among schools with matched test scores for 11 to 21 students (the 
smallest one-fourth of schools in our analysis), about 28 percent had 
inconclusive nce change scores, using the 0 nce standard. In contrast, 
among schools with matched scores for 59 or more students (the largest 
one-fourth of schools in our analysis), about 22 percent had inconclusive 



7 In our analysis, the range for a school with matched test scores for 60 students is + or - about 2 NCEs; 
for a school with matched scores for 15 students, the range is + or - about 4 NCEs (see app. HO- 
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nce change scores. 8 We found even greater differences at higher nce 
standards. (See fig. 2.2.) 



Figure 2.2: Schools With Matched Test 
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Change Scores 
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8 Among schools with matched scores for 10 or fewer students, using the 0 NCR standard, 56 percent 
would have inconclusive NCR change scores; using a 2 NCE standard, 64 percent; and using a 4 NCE 
standard, 68 percent Although we excluded such schools from our overall analysis, many may have 
been eligible for program improvement because they served more than 10 students during the school 
year. 
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Focus on Annual 
Changes in Test 
Scores May Neglect 
Needs of Schools With 
Lowest Achieving 
Students 



Focusing on annual changes in students* achievement-test scores may 
direct the attention of local and state officials away from schools whose 
students remain farthest below grade level but who meet the school 
district's standard for nce gains. Thus, the current identification process 
may not hold schools accountable for the Chapter 1 goal of helping 
students attain grade-level proficiency. The Department of Education and 
some state education officials have voiced concern that using annual 
change scores in judging Chapter 1 effectiveness is problematic because it 
does not consider the absolute level of performance of Chapter 1 students. 9 



We found that schools whose students had initially lower average 
achievement-test scores were less likely to be identified as needing 
program improvement than schools whose students had initially higher 
average test scores. 10 For example, under the 0 nce standard, about 
10 percent of schools with the lowest average initial test scores (less than 
32.1 nces) would be identified, compared with about 14 percent of schools 
with the highest average initial test scores (greater than 38.3 nces). 
Although this difference is not large, the difference between these groups 
is much greater when higher nce standards are used. (See fig. 2.3.) 



^ National Assessment of the Chapter 1 Program , U.S. Department of Education, p. 54. In response to 
this perceived problem, Connecticut has adopted an additional accountability standard, requiring all 
schools whose students average less than 32 NCEs in advanced reading or math skills to be identified 
for program improvement even if their students make annual gains. 

l0 Th\s may be due, in part, to a statistical phenomenon known as "regression to the mean," in which 
(1) schools with relatively high pretest scores will, on average, show lower gains than other schools 
and (2) schools with relatively low pretest scores will, on average, show higher gains than other 
schools — independent of program quality. 



9 
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Figure 2.3: Schools With Initially Lower H^^^^H^H 
Average Test Scores Less Likely to Be so p#rcant of schools 
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Pressure to Increase 
Test Scores May Have 
Negative Effect on 
Chapter 1 Instruction 



The high stakes placed on increasing students' achievement-test scores 
may have a negative effect on the instruction Chapter 1 students receive. 
Instruction may be narrowed, researchers have found, in response to 
pressure to increase achievement-test scores; 11 that is, subject areas 
covered on achievement tests tend to be taught to the exclusion of 
untested subject areas, and instruction tends to be oriented toward 
improving students 1 ability to recognize correct answers to multiple-choice 
questions rather than improving their higher order thinking skills. These 
practices are sometimes referred to as "teaching to the test." 



"See, for example, The Influence of Testing on Teaching Math and Science in Grades 4-12 , Center for 
the Study of Testing, Evaluation, and Educational Policy, Boston College, conducted for the National 
Science Foundation (SPA8954759, Oct. 1992); Kober, u The Role and Impact of Chapter 1, ESEA, 
Evaluation and Assessment Practices"; Mary Lee Smith, "Put to the Test: The Effects of External 
Testing on Teachers," Educational Researcher , Vol. 20, No. 5 (June-July 1991), pp. 8-11; Lauren B. 
Resnick and Daniel P. Resnick, "Assessing the Thinking Curriculum: New Tools for Educational 
Reform," prepared for the National Commission on Testing and Public Policy (Aug. 1989); and L. 
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In addition, focusing instruction on increasing students' achievement-test 
scores may not help Chapter 1 students succeed in the regular 
instructional program of the school district — one of the goals of Chapter 1. 
A criticism of current achievement tests is that they do not cover the 
content of regular classroom instruction. The Department of Education 
cautions, in its Chapter 1 Policy Manual, that gains on achievement tests 
may not translate into improved performance in regular classrooms. In 
response to concerns about current achievement tests, many states and 
districts are turning to performance assessment, testing that requires 
students to create products that demonstrate what they know and can do, 
such as a portfolio of writing samples, as opposed to answering 
multiple-choice questions. 12 

We found that some schools identified for the joint phase of program 
improvement simply re-targeted their current instructional practices on 
the subject areas in which test scores were too low, rather than making 
more comprehensive, long-term program changes involving new 
instructional approaches. For example, the principal of one school we 
visited said the school staff looked at students' test scores to determine 
which specific skills they needed to improve and then helped students 
master these skills by teaching to the test. The principal said this was not 
the most effective teaching approach but that schools do this because they 
are evaluated according to test scores. At another school we visited, the 
staff chose to spend more time teaching isolated skills covered on the 
achievement test used, the principal said, even though he considered this 
to be contrary to good instructional practice. Schools would do whatever 
is necessary to increase achievement-test scores, he said, because this is 
how schools are evaluated. 



Conclusions In judging program effectiveness, the Chapter 1 accountability system 

should rely less on achievement tests and more on holding schools directly 
accountable for meeting the three statutory goals of Chapter 1. Under the 
current system, states and districts may inaccurately judge the 
effectiveness of many schools' Chapter 1 programs. In addition, the 
current system may not provide an incentive for schools to adopt program 
changes directed at the Chapter 1 goals of helping children attain 



1 2 Testing in American Schools: Asking the Right Questions , U.S. Congress, Office of Technology 
Assessment, OTA-SKT-519 (Washington, D.C.: U.S. Government Printing Office, Feb. 1992), p. 201. We 
found that among the districts involved in the joint phase that also required the use of desired 
outcomes other than NCE gains, 14 percent required a desired outcome related to "samples or 
portfolios of student work" (see app. IX). 
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grade-level proficiency and succeed in the regular instructional program of 
the district. 

The accountability system could be improved by requiring school districts 
and states to (1) establish multiple desired outcomes that clearly relate to 
the statutory goals of Chapter 1 and (2) assess program effectiveness with 
multiple indicators of student performance. ThL c would provide a more 
complete and meaningful basis for deciding if a school should be identified 
for Chapter 1 program improvement. Districts and states could weigh 
evidence from a variety of indicators, such as performance on 
criterion-referenced tests, results of performance assessments, student 
grades, the length of time children remain in Chapter 1, or 
achievement-test scores — should these tests continue to be used, 

A requirement to use multiple outcomes and indicators, however, would 
make the law's local condition related to evidence other than 
achievement-test scores unnecessary. Leaving this local condition in the 
law would be confusing because considering other evidence would be 
required in making a determination about program effectiveness. 



RBCOmmCnd&tionS tO ^° " 1[l ^ rove ^ e identification process and to help reduce the emphasis 

placed on standardized achievement-test results, we recommend that the 

LilG COngrCSS Congress amend the Elementary and Secondary Education Act to require 

(1) states to establish, for schools' Chapter 1 programs, multiple desired 
outcomes related to the statutory goals of Chapter 1 and (2) districts to 
assess program effectiveness by considering whether evidence from 
multiple indicators of student performance shows substantial progress in 
achieving these outcomes. 

State education agencies should be required to specify, in their state 
program improvement plans, (1) the desired outcomes for Chapter 1 
schools, (2) the indicators that will be used to measure student progress 
toward those desired outcomes, (3) minimum standards for student 
performance on each indicator, and (4) a definition of substantial progress 
toward meeting the desired outcomes as a group (that is, how districts will 
weigh evidence from multiple indicators in judging whether their 
Chapter 1 schools arr effective). Districts should be allowed to set higher 
standards than required by their state education agency and to use, with 
the approval of their state agency, additional or alternative desired 
outcomes and indicators. 



g Page 31 GAO/HRD-93-69 Chapter 1 Accountability 



Chapter 2 

Chapter 1 Accountability System Hindered 
by Reliance on Achievement Testa 



The Congress should also remove from the program improvement 
provisions, the local condition that it can be argued, allows states and 
districts to disregard a school's achievement-test results if other indicators 
demonstrate that its Chapter 1 program is effective. This would prevent 
ambiguity about the identification process when multiple indicators of 
student performance are used to evaluate schools' Chapter 1 programs. 



Matter for 

Congressional 

Consideration 



To help ensure that states establish adequate standards for identifying 
Chapter 1 schools in need of improvement, the Congress should consider 
amending the Elementary and Secondary Education Act to require that the 
Secretary of Education review and approve the desired outcomes and 
indicators specified by states in their state program improvement plans. 
This review could focus on determining whether states have specified 
(1) desired outcomes and indicators that reflect high educational 
standards and pertain to the statutory goals of Chapter 1 and (2) a 
reasonable definition of substantial progress toward meeting multiple 
desired outcomes. 



Agency Comments 
and Our Evaluation 



In its February 26, 1993, comments on a draft of this report, the 
Department of Education agreed with our conclusion that using multiple 
measures of student performance would provide a better basis for 
deciding whether Chapter 1 schools are in need of program improvement. 13 
The Department expressed concern, however, that situations could arise 
in which one data source indicates program success while another 
indicates program failure. We agree that situations such as this will 
inevitably occur — as they do now, such as when a school falls short of the 
nce standard, but meets other desired outcomes. However, if our 
recommendations were implemented, state education agencies would be 
required to specify how evidence from all indicators will be considered 
together in evaluating Chapter 1 program effectiveness. State agencies 
could, for example, require schools to be identified for program 
improvement if they fail to meet two-thirds of the desired outcomes. State 
agencies could also place more emphasis on some desired outcomes than 
on others. 



The Department also cautioned that multiple measures will improve the 
identification process only if the measures used are valid and reliable. We 
agree that indicators used to measure Chapter 1 program effectiveness 
should reflect student success in meeting the statutory goals and yield 



ir, Scc appendix VIII for the Department's comments. 
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consistent results. Further, we believe the Department can play a valuable 
leadership role in assuring the quality of the desired outcomes and 
indicators used to judge Chapter 1 program effectiveness. If the Congress 
requires the Department to review and approve each state education 
agency's program improvement plan, as we suggest, this review could 
include consideration of the validity and reliability of proposed measures 
in each state's plan. 

Finally, the Department noted that our analysis did not empirically verify 
that multiple measures are more precise than a single measure for 
identifying Chapter 1 schools in need of program improvement. While this 
is true, we continue to believe — and the Department agrees — that the use 
of multiple measures will improve the identification process by providing 
a more complete and meaningful picture of program effectiveness. 
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Implementation of the joint phase of Chapter 1 program improvement was 
usually very similar to implementation of the local phase: The key players 
involved in developing plans and the strategies they chose changed little 
between the two phases. In both phases, school staff had the greatest 
influence in determining improvement needs and selecting improvement 
strategies. State education agency staff had considerably less influence on 
these activities than school and district staff, and state influence did not 
increase much in the joint phase. Some improvement strategies were used 
more often than others during both phases; the most commonly used 
strategies were increasing parental involvement and improving 
coordination between Chapter 1 and the regular instructional program. 

State education agencies provided more technical assistance to schools 
and districts during the joint phase than during the local phase. But this 
assistance was more often general, such as explaining the requirements of 
program improvement, than specific, such as helping individual schools 
develop their improvement plans; this assistance was also more often 
directed to districts than to individual schools. Specific state assistance, 
however, increased in the joint phase. In addition, the amount schools 
received was related to the number of staff that state agencies had to work 
with schools in the joint phase. 

School year 1991-92 was the first year that most states had schools in the 
joint phase of program improvement. Because of the limited experience 
that states, districts, and Chapter 1 schools have had with the joint phase 
of program improvement, it is too early to draw firm conclusions about its 
value or effectiveness. 



School Staffs Had 
Most Influence on 
Improvement Efforts 
During Both Phases 



School staffs were considered the most influential group in developing 
improvement plans in both the local and joint phases. Our questionnaire 
for district coordinators asked about the extent of influence various 
groups had in two key parts of plan development: determining schools* 
improvement needs and selecting improvement strategies. Almost 
80 percent of district coordinators rated school staffs as having a Very 
great" or "great" influence in determining improvement needs and 
selecting strategies during the joint phase. About 70 percent of district 
coordinators also rated district staff as having this degree of influence, but 
only about 40 percent rated state education agency staffs as having this 
much influence. (See fig. 3.1.) Generally, the greater the amount of state 
assistance a district received, the h ; gher the district coordinator rated the 
state staffs influence 
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Figure 3.1: School Staffs Had Most influence on Program Improvement Process 
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Influence on Determining Schools' Improvement Needs 



During Local Phase 
During Joint Phase 



Influence on Selecting Schools' Irnprr cement Strategies 



Notes: (1) Figure shows the percentage of district coordinators who rated each source as having 
•Very great" or "great" influence; (2) TAC = technical assistance center, R-TAC = rural technical 
assistance center. 



In the schools we visited, school, district, and state staffs influenced the 
program improvement process in different ways. Principals typically 
worked together with various school staff as a planning team to determine 
improvement needs and select improvement strategies. These teams 
usually focused on the subject areas and skills in which their Chapter 1 
students' achievement-test scores were low. Some schools also used a 
needs assessment instrument, provided by their state education agency or 
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a Chapter 1 technical assistance center, or surveys of parents or teachers. 
District staff usually provided schools with achievement-test data and 
pointed out which areas needed the most improvement. District staff also 
participated in school planning meetings and arranged staff development 
training. In three of the states in which we conducted case studies, staff 
from the state education agency had participated in on-site meetings, at 
one or both of the joint-phase schools we visited, to help develop joint 
plans. 



Schools Focused Most on 
Improving 

Achievement-Test Scores 
in Both Phases 



Schools' program improvement activities were most often aimed at 
improving Chapter 1 students 1 achievement-test scores. Our questionnaire 
for principals asked about which indicators of program effectiveness were 
first, second, and third most important for their schools to improve during 
each phase of program improvement. Principals rated students' 
achievement-test scores the most important indicator of program 
effectiveness during both phases (see fig. 3.2). In the joint phase, for 
example, about 82 percent of principals cited student performance on 
achievement tests as one of the three most important indicators for their 
schools to improve; 56 percent cited student grades; and 50 percent cited 
criterion-referenced test results. In contrast, only about 15 percent of 
principals cited the length of time students remain in Chapter I. 1 



^he indicators of program effectiveness that principals rated most important corresponded closely to 
the types of desired outcomes that districts used most often in identifying schools for program 
improvement For information on the types of desired outcomes districts used in school year 1990-91, 
see appendix IX. 
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Figure 3.2: Principals' Views of the ^^^^^H^H^H 

Most important Indicators of Program 100 f>«n»nt of Principals 
Effectiveness During Both Phases 
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Indicators of Program Eff«ctivanas« 



I 1 Local Phaso (N=878) 
Joint Phase (N=838) 

Note: Figure shows the percentage of principals who cited each indicator of program 
effectiveness as one of the three most important for their schools to improve. 



Schools in Joint Phase 
Usually Continued 
Improvement 
Strategies Begun in 
Local Phase 



Schools in the joint phase typically continued to use most of the same 
improvement strategies that they used in the local phase. In both phases, 
the strategies used most often were improving coordination between 
Chapter 1 and the regular instructional program and increasing parental 
involvement. In addition, large-city schools were much more likely than 
other schools to use certain strategies, especially adding an extended-day 
or summer program for Chapter 1 students. 

Most principals (67 percent) described the strategies their schools adopted 
in the joint phase as "exactly the same" or "very similar" to those they had 
adopted in the local phase (see fig. 3.3). While schools in the joint phase 
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sometimes dropped a strategy used in the local phase, schools were 
somewhat more likely to adopt a new one. About 53 percent of schools 
continued to use every one of their strategies from the local phase, 
compared with 47 percent that dropped at least one of their local-phase 
strategies. In addition, about 57 percent started at least one new strategy 
in the joint phase, compared with 43 percent that adopted no new 
strategies. School and district staff appeared to have confidence in the 
improvement strategies their schools had adopted in both phases. Few 
principals or district coordinators thought the strategies their schools used 
in the local phase had hindered school success; many more thought that 
the law allowed too little time to show improvement. 2 




The similarity between the local and joint phases is also reflected in the 
percentage of schools using particular strategies in the joint phase that 
these schools also used in the local phase. For every strategy listed in our 
questionnaire, more than half of the principals that reported using a given 
strategy in the joint phase also reported using it in the local phase; in most 
cases, more than 70 percent of the schools using a particular strategy in 
the joint phase were using it for the second time. (See fig. 3.4.) 



2 For more information on (he views of principals, district coordinators, and state coordinators on 
factors that hindered schools' success in the local phase, see appendix XI. 
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Figure 3.4: Most Schools That Used a Given Strategy in the Joint Phase Also Used It in the Local Phase 
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Some Improvement 
Strategies More Common 
Than Others 



Principals reported using some strategies much more often than others in 
both phases of program improvement (see fig. 3.5). The most common 
strategy in the joint phase was increasing parental involve ment, used by 
about 94 percent of schools; it was also the second most common strategy 
in the local phase, used by 86 percent of schools. The prevalence of this 
strategy may be due in part to its emphasis in the law: Districts are 
required to implement programs to increase parental involvement as a 
condition for receiving Chapter 1 fluids. 3 In striving to increase parental 
involvement during the joint phase, schools were also addressing the 
factor — insufficient parental involvement — that principals rated as the 
greatest hindrance to their schools' success during the local phase (see 
app. XI). Some examples of how the schools we visited were trying to 
increase parental involvement included (1) holding parent-teacher 
conferences at the school, (2) conducting home visits, (3) offering 
workshops to teach parents how they can help their children learn reading 



Public Law 100-207 section 1010(a)(2). 
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and math, and (4) assigning homework and providing instructional 
materials for parents and children to complete together. 



Figure 3.5: Some Improvement Strategies More Common Than Others During Both Phases 
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The second most common strategy in the joint phase was improving 
coordination between Chapter 1 and the regular instructional program , 
used by about 93 percent of schools; it was also the most common strategy 
in the local phase, used by 92 percent of schools. Improving coordination 
between Chapter 1 and the regular program is also mentioned prominently 
in the statute: To receive Chapter 1 funds, a district must assure, in its 
application to the state, that its Chapter 1 programs will "allocate time and 
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resources for frequent and regular coordination" of the Chapter 1 
curriculum with the regular instructional program. 4 Some of the schools we 
visited were trying to accomplish this by allotting time during the school 
day for planning meetings between Chapter 1 instructors and regular 
program teachers; 5 others required Chapter 1 instructors and regular 
program teachers to develop joint lesson plans. 

The third most common strategy during both phases was instructing 
Chapter 1 students on test-taking skills , used by about 83 percent of 
schools in the joint phase and 75 percent in the local phase. Instructing 
students on test-taking skills could represent the kind of improvement 
strategy the Congress did not intend for schools to adopt, because it does 
not involve a substantive change in the Chapter 1 program. If a significant 
amount of time was spent on this strategy rather than other instruction, 
the practice would be of dubious educational value. 6 However, instructing 
students on test-taking skills can also be seen as a way to level the playing 
field between students with more and less experience taking 
multiple-choice achievement tests; 7 this instruction may be an attempt to 
ensure that students do not score lower than they are capable of scoring 
because they misunderstood the test format, or instructions. As long as 
schools are held accountable for student performance on multiple-choice 
achievement tests, it is likely that teachers will continue to spend time 
instructing students on test-taking skills. 

The fourth most common strategy in both phases was adopting a new 
instructional approach for Chapter 1 . This strategy was used by about 
73 percent of schools in the joint phase and 66 percent in the local phase. 
Some examples of new instructional approaches for Chapter 1 adopted by 
the schools we visited included (1) cooperative learning, in which students 
work together in small, mixed-ability groups to help one another learn, and 
(2) Reading Recovery, a program in which expert teachers provide first 



'Public Uw 100-207 section 1012(c)(3). 

^The extent to which planning meetings alone will bring about needed improvements in program 
coordination is unclear, according to experts. Literature on program coordination suggests that further 
actioas by principals and district coordinators may be needed to better align curriculum and 
instruction between Chap' < r 1 and the regular instructional program. For a review of this literature, 
see Richard L Allingtnn and Peter Johnson, "Coordination, Col Ial oration, and Consistency: The 
Redesign of Compensatory and Special Education Interventions," in Effective Programs for Students at 
Risk, edited by Robert K. Stavin and others (Needham Heights, Mass.: Allyn and Bacon, 1989), 
ppT320-54. 

•Our survey of principals did not ask aUmt the amount of time schools spent on this or other 
improvement strategics. 

7 Education Reform: Initial Effects in Four School Districts (GAO/PKMI)^-^, Sept. 26, 1989), p. 38. 
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graders with intensive, one-on-one reading instruction, to bring their skills 
quickly up to the level appropriate for their age. 



Large-City Schools Used During both phases of program improvement, large-city schools, which 

Some Strategies More serve Wgh concentrations of disadvantaged children, 8 used some strategies 

Often Than Other Schools more often than schools 5n other locations. In the joint phase, for example, 

62 percent of large-city schools added summer programs for Chapter 1 
students, compared with 22 percent of schools located in other areas (see 
fig. 3.6). In addition, 53 percent of the schools in large cities added an 
extended-day program to provide Chapter 1 services before or after 
school, compared with 19 percent of all other schools. These differences 
appear to show that many large-city schools are responding to the needs 
of their Chapter 1 students by providing more minutes of instruction per 
day and more continuous education year-round. 



ERLC 



*Eighty-four percent of large-city schools in the joint phase are high-poverty schools (defined as having 
75 percent or more of their students participating in the free or reduced -price lunch program). 
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Figure 3.6: Schools In Large Cities ^^^^^^^^^^■^^^^^^■^ 
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Note: Large cities are defined as areas with a population of 400,000 or more or with a population 
density of at least 6,000 people per square mile. 



State Assistance 
Increased During the 
Joint Phase 



State education agencies provided more technical assistance to districts 
and schools during the joint phase than in the local phase. In both phases, 
however, this assistance addressed general program requirements more 
than the improvement needs of specific schools; this assistance was also 
directed more to district officials than to school officials. The amount of 
specific technical assistance that states provided was related to the 
number of staff the state agencies had to work with schools in the joint 
phase. 
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District coordinators and principals received more technical assistance 
from their state education agencies during the joint phase than during the 
local phase. We asked about three types of technical assistance in our 
questionnaires: (1) explaining the requirements of program improvement, 
in general; (2) providing general information about improving schools; and 
(3) providing specific assistance on improving individual schools. For each 
of these three types of assistance, the percentage of district coordinators 
and principals reporting a "very great" or "great" amount was higher in the 
joint phase than in the local phase. The type of technical assistance that 
increased the most was specific assistance on improving individual 
schools. Despite this increase, however, more than one-third of schools 
and about one-fifth of d : stricts still reported a "very smair or u no w amount 
of specific technical assistance from their state education agency during 
the joint phase. (See figs. 3.7 and 3.8.) We also found that principals and 
district coordinators who received little or no specific assistance were 
much less satisfied with the amount and types of state assistance than 
those who received a greater amount of specific assistance. 
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Figure 3.7: Amount and Types of 
Technical Assistance That District 
Coordinators Reported Receiving 
From the State Education Agency 



Parcant of CHatrict Coord! nators 
66 




"Vary 
Small" 
or or 
"Graat" "Nona" 




"Vary 
Small" 
or or 
"Graat" "Nona" 




"Vary "Vary 

Graat" Small" 

or or 

"Great" "Nona" 



Amount of Tachnical Aasistanca 



General 
Explanation of 
Program 
Improvement 
Requirements 



~1 Local Phase 
H Joint Phase 



General 
Information 
on Improving 
Schools 



Specific 
Information 
on Improving 
Individual 
Schools 



ERLC 



Page 45 



4,; 

GAO/IIRD-93-69 Chapter 1 Accountability 



Chapter 3 

Implementation of the Joint Phase of 
Program Improvement Similar to 
Implementation of the Local Phase 



Figure 3.8: Amount and Typ*s of 
Technical Assistance That Principals 
Reported Receiving From the State 
Education Agency 
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In both phases of program improvement, state education agencies 
provided higher levels of general assistance than specific assistance. 
During the joint phase, 66 percent of district coordinators reported a Very 
great" or "great" amount of assistance involving a general explanation of 
program improvement requirements, compared with 46 percent that 
reported this amount of specific assistance on improving individual 
schools (see fig. 3.7). Among principals, 43 percent reported a "very great" 
or "great" amount of assistance involving a general explanation of program 
improvement requirements, compared with 31 percent that reported this 
amount of specific assistance on improving their own schools (see 
fig. 3.8). 
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State education agencies also appear to have targeted their technical 
assistance more often at the district level than directly to individual 
schools. In both phases, district coordinators reported higher levels of 
each type of technical assistance than did principals (compare figs. 3.7 and 
3.8). 



Amount of Specific 
Technical Assistance to 
Schools Related to State 
Staff Capacity 



The amount of specific technical assistance state education agencies 
provided to schools in the joint phase was related to the number of 
full-time equivalent (fte) state staff available to work with such schools. In 
states with fewer than 10 joint-phase schools per fte state staff, these 
schools received significantly more specific assistance than did joint-phase 
schools in states with more than 20 schools per fte. 9 (See fig. 3.9.) Several 
state Chapter 1 coordinators we interviewed were concerned about their 
agencies' ability to provide districts and schools with adequate technical 
assistance during the joint phase. Mississippi, for example, had only two 
fte staff to serve 282 joint-phase schools. 10 



»A significant relationship did not exist, however, between the amount of specific assistance state 
agencies provided to districts and the number of FTE state staff available to work with them. 

'"Mississippi's stale Chapter 1 coordinator said tliat the staff shortage was due to a statewide hiring 
freeze applied across the board, regardless of source of funds; the hiring freeze prevented h.m from 
hiring additional staff even though fe<lentl Chapter 1 funds were available tn do so. 
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Figure 3.9: Amount of Specific 
Technical Assistance That Principals 
Reported Was Related to State Staff 
Capacity 
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Many more schools will enter the joint phase during the 1992-93 school 
year, further taxing the ability of some states to provide assistance in 
developing and implementing joint improvement plans. Some state 
education agencies could benefit from increased staff capacity to meet this 
increased need. One way to increase state capacity, cited by some state 
Chapter 1 coordinators, would be to allow states to use a portion of 
federal Chapter 1 program improvement funds 1 1 without obtaining prior 
approval from districts and schools, as currently required by law. 
However, the state role during the joint phase is relatively new and still 
evolving, and some states have added staff to work with schools in 
program improvement even without added flexibility in the use of program 
improvement funds. Therefore, it is unclear whether federal action to 
provide states with greater flexibility in the use of these funds is needed at 
this time. 



"These funds, which totalled $14.8 million in fiscal year \UU2, are separate from regular Chapter 1 
allocations. States typically distribute these funds to districts in the form of small grant* In school 
year 1900-01, the median grant size, among those districts that received a grant, was $2,000 per district 
(Source: Millsap and others, The Chapter 1 Implementation Study.) 
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Chapter 3 

Implementation of the Joint Phase of 
Program Improvement Similar to 
Implementation of the Local Phase 



Because of the limited experience that states, districts, and Chapter 1 
schools have had with the joint phase of program improvement, it is too 
early to draw firm conclusions about its value or effectiveness. We 
conducted our study during the first school year in which most states had 
Chapter 1 schools in the joint phase. In subsequent years, the joint phase 
may be implemented differently. For example, as district and school 
officials become more familiar with the requirements of the joint phase, 
state officials may be able to spend less time providing general 
information and more time assisting individual schools with their 
improvement efforts. This will depend, to some extent, however, on state 
Chapter 1 coordinators having sufficient staff numbers and expertise. 

The role and influence of state staffs and the effectiveness of the joint 
phase may be important in terms of improving Chapter 1 programs; the 
joint phase may also be instructive for policymakers as they design other 
accountability systems and determine what role state education agencies 
should play. As the nation moves toward holding schools more 
accountable for student outcomes, Chapter 1 program improvement 
represents one model for doing so. Likewise, the joint phase represents 
one model for state involvement in school accountability efforts. 
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School, District, and State Officials' Views 
on the Accuracy of the Program 
Improvement School Identification Process 



Substantial portions of the principals, district Chapter 1 coordinators, and 
state coordinators we surveyed thought the identification process was 
somev/hat inaccurate. This may be important because a recent study 
found that when local officials believed schools had been wrongly 
identified for program improvement, officials sometimes delayed 
development of improvement plans and, in some cases, undertook no 
improvement activities at all. 1 



A majority of principals thought their schools had been accurately 
identified to be in the joint phase during school year 1991-92, but a 
substantial number thought otherwise. Among principals, about 63 percent 
thought their schools had been accurately identified to be in the joint 
phase of program improvement during school year 1991-92; about 
30 percent thought their schools had not been accurately identified; and 
7 percent answered "don't know." 

We found that principals' views on whether their schools had been 
accurately identified were related to the number of indicators on which 
the identification decision had been based. Principals whose schools were 
identified because of both insufficient nce gains and lack of substantial 
progress toward other desired outcomes were more likely to believe their 
schools had been accurately identified than those whose schools were 
J identified on the basis of nce gains alone or other desired outcomes alone 
(see fig LI). 



Principals' Opinions 
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'Millsap and others, The Chapter 1 Implementation Study . 
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Figure 1.1 : Principals Whose Schools ■^■■■^^^^M 

Were Identified for the Joint Phase by 100 Percent of Principals 

NCE Gains and Other Desired 

Outcomes Were Most Likely to Say 90 

School Was Accurately Identified 80 



70 

63 




Basis on Which School Was identified for Joint Phase 



In explaining why they thought their schools had been inaccurately 
identified, principals were most likely to challenge the legitimacy of using 
achievement tests to gauge Chapter 1 program effectiveness. To 
understand why some principals thought their schools had been 
inaccurately identified, our questionnaire presented three possible reasons 
and asked principals to check all that applied. Sixty-three percent agreed 
that achievement tests w do not indicate the effectiveness of our Chapter 1 
program;" 57 percent agreed that "our school was identified because we 
failed to meet the nce standard, regardless of other evidence of our 
effectiveness;" and 54 percent agreed that "the nce scores of a few 
students put our school in joint program improvement." 



TV t * t j q t t A majority of district and state Chapter 1 coordinators thought the process 

District anQ oiaie they use(J l0 identify schools for program improvement was accurate most 

Coordinators' of the time. About 36 percent of district coordinators and 31 percent of 

Opinions state coordinalors said their identification process was "always" or 
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"almost always" accurate; in addition, 25 percent of district coordinators 
and 37 percent of state coordinators rated their identification process 
"more often accurate than inaccurate." However, a substantial proportion 
of each group — 40 percent of district coordinators and 32 percent of state 
coordinators — thought the identification process they used was a as often 
accurate as inaccurate" or inaccurate most of the time. (See fig. 1.2.) 



Figure 1.2: District and State Coordinators' Views on the Accuracy of the Identification Process 



50 Parcent of District and State Coordinators 
45 




District and state coordinators who felt their identification process was 
not always accurate had different views about the impact of such 
inaccuracies. District coordinators were much more likely to believe that 
some schools had been inappropriately identified for the joint phase than 
they were to believe that any schools had, inappropriately, not been 
identified. About 44 percent of these district coordinators indicated that 
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one or more schools had been identified for the joint phase that, in their 
opinion, did not need to be in program improvement; only about 7 percent 
indicated that some schools had not been identified for the joint phase 
that, in their opinion, did need to be in program improvement. State 
coordinators, however, were more likely to believe that schools had been 
inappropriately excluded than inappropriately included in the joint phase. 
About 20 percent said that some schools had been identified for the joint 
phase that did not need to be in program improvement But about 
28 percent said that some schools had not been identified for the joint 
phase that did need to be in program improvement 

We found that district coordinators' views on the accuracy of the 
identification process were related to the number of desired outcomes 
used to judge Chapter 1 program effectiveness. Coordinators from districts 
that had established additional desired outcomes viewed the identification 
process as more accurate than those from districts that used only average 
nce gains to judge program effectiveness (see fig. 1.3). We also found that 
the higher the minimum standard for average nce gains, the more accurate 
district coordinators saw the identification process (see fig. 1.4). 
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Figure 1.3: District Coordinators Were 
More Likely to Consider the 
identification Process Accurate If Their 
Districts Required the Use of Desired 
Outcomes Other Than NCE Gains 



75 Psrcent of District Coordinators 




Required Not Required 

(N=238) (N«163) 

District Policy on Other Desired Outcomes 




Generally Accurate 

About As Accurate as Inaccurate 

Generally Inaccurate 



Notes: (1) "Generally accurate" includes the responses "always accurate." "almost accurate," and 
"more often accurate than inaccurate;" (2) "Generally inaccurate" includes the responses "more 
often inaccurate than accurate," "almost always inaccurate," and "always inaccurate." 
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Figure 1.4: the Higher the NCE i^^^^^^^^^^^^" 

Standard Used in a District, the More 75 Psrcsnt of District Coordinators 



Likely the District Coordinator Was to 
Consider the Identification Process 
Accurate 




0 NCES 1 NCES 2 NCES 3 NCES 

(N=»9) (N=87) (N=111) (N=80) 

Minimum Stsndard for NCE Gains Ussd In District 

Note- Figure shows the percentage of district coordinators who rated the identification process 
"always accurate," "almost always accurate." or "more often accurate than inaccurate." 



Among state coordinators, however, those from states that required the 
use of additional desired outcomes did not see the identification process 
as more accurate than those from states that only require the use of NCE 
change scores to identify schools. In addition, there was no clear 
relationship between state minimum nce standards and state coordinators' 
views on the accuracy of the identification process. 

In explaining why they thought the identification process was inaccurate, 
both district and state Chapter 1 coordinators were most likely to express 
concern that a school's nce change scores can be affected by the test 
scores of just a few students. Our questionnaires presented four possible 
reasons and asked district and state coordinators to indicate the extent to 
which each of these reasons was responsible for inaccuracies. About 
85 percent of district coordinators and 52 percent of state coordinators 
responded that to a "very great" or "great" extent, the identification 
process was inaccurate because "the nce scores of a few students can put 
a school in or out of program improvement." (See table LI.) 
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Reasons Cited by District ^^^■■^■^^^^^^VH^^^^^^^^^B^^^^H^^^^^^^^^^H 

and State Chapter 1 Coordinators for Percentage saying this was a reason, 

Inaccuracies in identification Process to a "very great" or "great" extent, 

for inaccuracies In the identification 



process 


Reasons for inaccuracy 


District 
coordinators' 


State 
coordinators* 


School identification is based on only 1 year of 
data 


56 


43 


Norm-referenced tests do not indicate 
Chapter 1 program effectiveness 


69 


35 


Schools are identified if they fail to meet the 
NCE standard, regardless of other evidence of 
their effectiveness 


79 


35 


The NCE scores of a fc ,v students can put a 
school in or out of program improvement 


85 


52 



a Number of respondents on individual items ranged from 322 to 336. 



b Number of respondents on individual items ranged from 48 to 49. 
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Number and Characteristics of Schools and 
Districts in the Joint Phase During 
School Year 1991-92 



About 1,400 Chapter 1 schools in about 500 districts were identified for the 
joint phase of program improvement during school year 1991-92. The 
number of joint-phase schools and districts in each state in school year 
1991-92, as well as the number of schools and districts that provided 
Chapter 1 services during the preceding school year, 1990-91, is shown in 
table ILL 



Table 11.1: Schools and Districts in the 
Joint Phase of Program Improvement 
by State 



ERLC 



Do 



State 


Chapter 1 
schools in 
state, school 

vpar 1 QQH-Q1 


onapter i 
schools in 
joint phase, 
school year 
1 991 -92 


Chapter 1 
districts in 
state, school 
vear 1 990-91 


onapier i 
districts in 
joint phase, 
school year 
1991-92 


Alabama 


934 


13 


128 


10 


Alaska 


199 


7 


52 


3 


r\\ 1 Z_WI 1CX 


525 


28 


209 


13 


Arkan^a^ 


885 


95 


316 


53 


California 


3,880 


a 


742 


b 


Colorado 


581 


11 


175 


6 


Connecticut 


636 


1 


165 


1 


Delaware 


110 


3 


19 


2 


District of 
Columbia 


104 


0 


1 


0 


Florida 


1,051 


18 


67 


6 


Georgia 


1,078 


9 


184 


9 


Hawaii 


88 


28 


7 


7 


Idaho 


527 


11 


109 


8 


Illinois 


2,351 


182 


954 


12 


Indiana 


1,157 


35 


293 


13 


Iowa 


1,007 


0 


430 


0 


Kansas 


739 


4 


302 


3 


Kentucky 


1,054 


12 


175 


11 


Louisiana 


851 


38 


66 


17 


Maine 


557 


4 


168 


2 


Maryland 


435 


69 


24 


11 


Massachusetts 


1,012 


6 


325 


1 


Michigan 


2,073 


101 


562 


43 


Minnesota 


941 


4 


427 


1 


Mississippi 


779 


282 


159 


84 


Missouri 


631 


3 


482 


2 


Montana 


581 


3 


313 


2 


Nebraska 


558 


3 


322 


3 
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OldlC 


Chapter 1 
schools In 
state, school 

y ecu i i 


Chapter 1 
schools in 
joint phase, 
school year 


Chapter 1 
districts In 
state, school 

Jval I9JU <7 1 


Chapter i 
districts in 
joint phase, 
school year 
iggi.g2 


iNcvaua 


QQ 


5 


16 


3 


INCW I (CM 1 l^/ol IN C 


330 




159 


1 


Mpw . Iptqpv/ 


2,038 


8 


592 


4 


Mpw Mpyiph 


400 


10 


88 


4 


Mpw York 


3,052 


6 


719 


3 


Nnrlh Carolina 


1,341 


25 


134 


11 


North Dakota 


333 


o 


255 


o 


Ohio 


2,015 


1 


613 


1 


Oklahoma 

\_/f\ICll 1 Wl I Id 


1 ,1 19 


3 


568 


3 


Orpnon 


790 


1 


298 


1 


Ppnn^vk/a nia 


2,312 


28 


500 


9 


Rhorlp l^lanri 


154 


3 


37 


2 


South Carolina 


540 


31 


91 


11 


South Dakota 


432 


2 


177 


2 


Tennessee 


908 


99 


139 


17 


Texas 


b 


75 


998 


38 


Utah 


275 


6 


40 


3 


Vermont 


299 


7 


61 


5 


Virginia 


826 


61 


135 


23 


Washington 


990 


3 


284 


3 


West Virginia 


558 


47 


55 


18 


Wisconsin 


1,379 


21 


422 


17 


Wyoming 


119 


2 


46 


2 


Total 


45,633 


1,415 


13,603 


504 



a Although California reported 356 schools in the joint phase, we do no* include them here 
because the state education agency considers all schools identified for program improvement to 
be in the joint phase; California schools do not go through a local phase. 



information not available. 



The exact number of joint-phase schools and districts is difficult to 
determine, however, because (1) state coordinators could not always 
provide us with definitive mailing lists and (2) many of the principals and 
district coordinators we surveyed claimed that their schools were not in 
the joint phase. In some cases, the mailing lists that state coordinators 
provided us with did not represent the final number of districts and 
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schools involved in the joint phase; even in March 1992, with the school 
year more than half over, some state coordinators could not tell us exactly 
which schools were in the joint phase. A common reason was that states 
were still awaiting, or had not yet finished analyzing, test score data from 
districts using a fall-to-fall testing cycle. 

Despite being included on their state's mailing list of joint-phase schools 
and districts, many principals and district coordinators who returned a 
questionnaire indicated that they were not involved in the joint phase of 
program improvement. We received valid responses from 478 district 
coordinators and 1,199 principals. 1 However, about 13 percent of the 
district coordinators said no schools in their district were currently in the 
joint phase of program improvement. In addition, about 18 percent of the 
principals said their schools had not been identified for the joint phase 
during school year 1991-92, and 4 percent said they did not know whether 
or not their schools were in the joint phase. 

We did not systematically investigate the reasons why some principals and 
district coordinators said they were not involved in the joint phase; 
however, we have some anecdotal evidence. One district coordinator, for 
example, notified us that her district was appealing the state's decision to 
identify one of its schools for the joint phase, claiming this decision was in 
error; when we later received this district coordinator's questionnaire, it 
indicated that no schools were in the joint phase. Another district 
coordinator, from a large urban district in which many principals claimed 
not to be in the joint phase, said that some of these principals did not 
understand the program improvement process, even though he had 
explained the joint phase in several meetings. In a few cases, officials said 
that a school was closing down or not offering Chapter 1 services for the 
1991-92 school year; thus, although these schools had failed to improve 
sufficiently during the local phase, they would not be entering the joint 
phase. In some cases, the state coordinators included a school on their 
joint-phase mailing lists by mistake. Finally, it is possible that some 
principals and district coordinators were mistaken when they indicated 
they were not involved in the joint phase. 

Our questionnaires for principals and district coordinators included a 
variety of questions intended to gather basic descriptive information about 
schools and districts involved in the joint phase of program improvement. 
In the average joint-phase school, 36 percent of the students were served 



'We defined a valid response as one that included an answer for the first question, which asked about 
whether the school or district was in the joint phase. 
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by Chapter 1. In addition, most of these schools had a substantial 
proportion of students from economically disadvantaged families; on 
average, 64 percent of the students in these schools participated in the free 
or reduced-price lunch program. The distribution of joint-phase schools by 
poverty level is shown in table H.2. 



Table 11.2: Distribution of Schools in 
the Joint Phase Among Different 
Levels of Poverty 



Table 11.3: Distribution of Schools in 
the Joint Phase Among Different 
Locations 



Poverty level in school 


Percentage of schools 
(N=827) 


High 

(75 percent or more of students in free or reduced-price 
lunch program) 




40 


Medium 

(50 to 74 percent of students in free or reduced-price lunch 
program) 




30 


Low 

(0 to 49 percent of students in free or reduced-price lunch 
program) 




30 


Total 




100 


Joint-phase schools were most often located in rural areas, followed by 
small towns, mid-size cities, and large cities (see table IL3). 




Location of school 


Percentage of schools 
(N=896) 


Large central city 




15 


Urban fringe of large city 




5 


Mid-size central city 




17 


Urban fringe of mid-size city 




7 


Large town 




3 


Small town 




23 


Rural area 




29 


Total 99 



Note: Numbers do not total to 100 due to rounding. 

Source: GAO analysis of valid responses to survey of joint-phase school principals, using the 
location codes from the Department of Education's Common Core of Data Public School Universe 
(1988-89). 



Among the principals who answered our questions about program 
improvement funding, more than half reported that their school had been 
allocated no funds for program improvement in school year 1991-92; more 
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than half also reported no such funds during the prior school year, 
1990-91. Among those who reported receiving some program improvement 
funds in school year 1991-92, the median amount was $2,000; for school 
year 1990-91, it was $1,359. 

Additional descriptive information is provided in appendixes IV, V, and VI, 
which present aggregate responses to all questions in our state, district, 
and school questionnaires, respectively. 
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Based on our review of the literature and interviews with local, state, and 
federal education officials, we had several specific concerns about the 
accuracy of the identification process when only nce change scores are 
used to evaluate schools for program improvement. One of our primary 
concerns was that some schools may be identified — and others not 
identified — on the basis of scores that do not constitute strong statistical 
evidence for deciding whether or not the school met the standard for nce 
gains. We were also concerned that small schools might be more likely 
than large schools to have such statistically inconclusive nce change 
scores. Finally, we were concerned that schools with the lowest achieving 
students might be identified for program improvement less often than 
schools with higher achieving students. To demonstrate the extent to 
which these problems occurred, we analyzed a data set containing 
achievement-test scores for Chapter 1 students in one large state. This 
appendix describes the technical details of our analysis. 1 



Data Source and 
Scope of Analysis 



We obtained a data set that initially contained achievement-test scores in 
reading, math, or language arts for 165,707 students in 2,475 Pennsylvania 
schools. This data set, which included test scores from school years 
1989-90 and 1990-91, is the same data set that Pennsylvania's state 
education agency uses to track the progress of its Chapter 1 students and 
schools. We then used several criteria to limit the scope of our analysis to 
include only certain students and schools, as well as certain types of 
achievement-test scores. These criteria and our rationale for applying 
them are described below. 



First, we limited our analysis to include only students' achievement-test 
scores for reading. We decided to use only reading test scores primarily 
because, in the Pennsylvania data set, far more schools had students with 
achievement-test scores for reading than for math. 

Second, we further limited our analysis to include only students' test 
scores for advanced skills. We chose to focus on advanced reading skills 
primarily because we could easily determine the scores used to measure 
advanced skills, but not the scores used to measure basic skills. Advanced 
skills in reading are measured by students' scores on the reading 
comprehension portion of an achievement test. In contrast, basic skills in 
reading may be measured by students' scores on at least two different 



! We modeled our analysis on a study by Alan Davis of the University of Colorado at Denver, who 
served as a consultant for our study. See Alan Davis, "lipping the Stakes; I'sing (iain Scores to Judge 
Local Program Effectiveness in Chapter 1," Educational Evaluation and Policy Analysis , Vol. 13, No. 4 
(Winter 1991), pp. 980-88. 
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portions of an achievement test — for example, vocabulary or word 
analysis subtests — or by students' total scores in reading. 

Third, we included only students in grades 242 because, for children 
below second grade, Chapter 1 policy guidelines prohibit schools from 
using achievement-test scores for program improvement purposes. 

Fourth, we included only students with matched test scores in a particular 
school because schools' nce change scores are supposed to represent the 
average difference in the same students' achievement-test scores from one 
year to the next We also made sure that students' matched scores were 
obtained over a 1-year period; that is, students were tested on either a 
fall-to-fall or spring-to-spring cycle. 

Fifth, we limited our analysis to include only schools with matched test 
scores for more than 10 students. Schools that serve 10 or fewer students 
in Chapter 1 during an entire school year are exempted by law from 
consideration for program improvement. 

After applying all of the above criteria and eliminating records with 
duplicate identification numbers and invalid test scores, the final data set 
we used in our analysis contained information on 106,825 Chapter 1 
students in 2,115 schools. In these schools, the average Chapter 1 student 
pretest score was 35.1 nces and their average posttest was 40.2 nces; thus, 
the average nce change score in these schools was 5.1 nces. The median 
number of Chapter 1 students per school in our analysis was 36. 



Mpth nr\n\r\0\7 ^ ne of our P rimar y methods for demonstrating the extent of imprecision 

lVieiilUUUlUgy in the j(j en tjfication process involved constructing confidence intervals 

around schools' nce change scores. We used confidence intervals as a 
means of estimating the extent to which schools' nce changes scores were 
affected by measurement error (explained below). 



Calculating NCE For each student in our data set, we computed the change in his or her 

Change Scores achievement-test scores from one year to the next by subtracting the 

pretest score from the posttest score. Then we computed the average of 
these changes for ail the students in each school. These averages 
constituted the schools' nce change scores. 
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Estimating Measurement One problem affecting the use of nce change scores to identify Chapter 1 
Error schools for program improvement is measurement error. 2 Individual 

students' scores on achievement tests are not completely reliable; on 
repeated testing these scores would be subject to a small degree of 
random fluctuation. In addition, individual students* change scores are less 
reliable than either their pretest or posttest scores because change scores 
combine the measurement errors associated with both tests. Similarly, the 
nce change score for a school is less reliable than the average nce score 
for its students at one point in time. Thus, a school's nce change score is 
an imprecise measure of its "true" nce change score — the hypothetical 
score that would be obtained if pretest and posttest scores were 
completely reliable, that is, if these scores were not subject to random 
variation because of measurement error. 

To assess the accuracy of the identification process, we accounted for the 
measurement error associated with schools' nce change scores. To do this, 
we estimated the overall variance in students' change scores due to 
measurement error. In developing this estimate, we first calculated the 
reliability of students' change scores, based on tne following formula: 



(r x *c£) + (r y *c 2 y ) - (2*r xy *G x *G y ) 
<*l + <*y -(2*r jy *a x *a y ) 

where 



r x = pretest reliability (.78), 

a 2 x = variance of pretest scores (148.6), 

r y = posttest reliability (.84), 

a 2 y = variance of posttest scores (202.8), 

= correlation between pretest and posttest scores (.41), 
a x = standard deviation of pretest scores (12.2), and 
a y = standard deviation of posttest scores (14.2). 

Variances, standard deviations, and correlations were calculated from the 
test scores of all students from all schools in the data set. Pretest and 
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2 Anothor problem is known as "regression to the mean.'* This term refers to a statistical phenomenon 
in which (1) schools with relatively high pretest scores will, on average, show lower gains than other 
schools, and (2) schools with relatively low pretest scores will, on average, show higher gains than 
other schools— independent of program quality. To simplify our analysis, we did not adjust, for 
regression to the mean. 
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posttest reliability coefficients were based on a formula for estimating 
reliability in a restricted population. 3 

The result of this formula indicated that the reliability of individual 
students' change scores was .69. From this reliability coefficient we 
estimated that about 31 percent (1.00 - .69) of the variance in individual 
students' change scores was attributable to measurement error. Finally, 
since the variance in individual students' change scores in our analysis 
was 207.36, we estimated the overall variance due to measurement error 
as 64.3 (.31 x 207.36). We then used this estimate in developing confidence 
intervals around each school's nce change score. 



Constructing Confidence We constructed confidence intervals around each school's nce change 
Intervals score t0 reflect the amount of measurement error associated with that 

score. 4 These confidence intervals were based on the following standard 

equation: 



X _ z * 



a" 
~7F 



The components of this equation are described below: 

X: This term represents the nce change score for the school. 

z: This term reflects the degree of confidence we wanted to have that a 
schoors "true" nce change score falls between the upper and lower 
bounds of the confidence interval. By losing a value of 1.96 for z, we 
developed confidence intervals at the 95 percent level. This means that we 
can be 95 percent confident that a school's true nce change score falls 
within the confidence band. 

a 2 : This term represents the overall variance in individual students' change 
scores that is attributable to measurement error (64.3), as explained 
earlier. 



*See H. Guilikson, Theory of Mental Tests (Hillsdale, N..I.: Erlbaum, 1AS7), p. 1 11. The components of 
this formula include the overall variance of NCE scores in the full population (113.52), the average 
internal consistency estimate of reliability for the achievement tests most commonly used by the 
schools in our analysis (.03), and the variance of the pretest scores (148.0) or posttest scores (202.8). 

4 While confidence intervals are most commonly used to reflect the error attributable to making 
estimates about a population from a sample, the confidence intervals we constructed reflect 
measurement error arising from standardized achievement tests rather than from sampling error. 
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n: This term represents the number of students on which the school's nce 
change score is based. 

Because a 2 is held constant for all schools in our analysis, the difference in 
the size of schools' confidence intervals is determined solely by the 
number of students contributing to a school's nce change score (n). As 
shown in fig. HI. 1, the width of the confidence interval decreases as the 
number of students contributing to the nce change score increases. For 
example, if a school had matched test scores for only 15 students, the 
confidence interval for its nce change score would be + or - about 4 nces; 
if a school had matched test scores for 60 students, the confidence interval 
for its nce change score would be + or - about 2 nces. 



Figure HI.1: Confidence Intervals Become Narrower as the Number of Students With Matched Test Scores Increases 

Size of Confidence interval 
7.5 




5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 

Number of Chapter 1 Studentt With Matched Test Scores in School 



GAO's Analysis We performed three analyses using our data set; these corresponded to the 

three concerns outlined at the beginning of this appendix. We conducted 
each analysis using three different standards for nce gains (0 nces, 2 nces, 
and 4 nces), to reflect the range of nce standards used in different states 
and districts around the country. 
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First, we dete mined the number of schools with nce change scores that, 
because of measurement error, would not permit a confident judgement 
about whether students actually exceeded or fell short of various 
standards for nce gains. If the nce standard fell within the range of the 
confidence interval, we classified the school's nce change score as 
inconclusive . For these schools, we could not be confident about whether 
the true nce change score was above or below the standard. Conversely, if 
the nce standard did not fall within the range of the confidence interval, 
then we classified a school's nce change score as conclusive . For these 
schools, we could be 95 percent confident that the true nce change score 
was either above or below the standard 

Second, we examined the extent to which schools with matched test 
scores for small numbers of students were more likely to have 
inconclusive nce change scores than schools with matched scores for 
large numbers of students. We divided the schools in our data set into four 
approximately equal-sized groups, based on the number of students with 
matched test scores. We then calculated the percentage of each group with 
inconclusive nce change scores. 

Third, we estimated the extent to which schools with initially high average 
achievement-test scores were more likely to be identified than schools 
with initially low average test scores. We divided the schools in our data 
base into four approximately equal-sized groups, based on their Chapter 1 
students' average pretest scores. Then we calculated the percentages of 
each group that had nce change scores above and below the standard for 
nce gains. 
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U.S. GENERAL ACCOUNTING OFFICE 
JOINT PROGRAM IMPROVEMENT STUDY 
Survey of State Chapter 1 Coordinators 


INTRODUCTION 


Please provide the name of the one person 




responsible for completing this questionnaire so that 


The U.S. General Accounting Office, an agency of 


we may call and clarify information, if necessary. 


the Congress, is conducting a study of state and 




local implementation of the Chapter 1 program 


Names: 


improvement provision. As part of our study, we 




are sending this questionnaire to all State Education 


Title: 


Agencies (SEAs) in the nation. 






Offices Phones: 


The primary purpose of this questionnaire is to 




obtain information on states' experiences with the 


State: 


loini SEA/LEA phase of program improvement. 




Schools enter the loint phase if they are identified 




as needing improvement after their local plan has 


LEA AND SCHOOL INFORMATION, 


been in effect for one full school year. 


SY 1990-91 




1. During school year (SY) 1990-91, how many 


INSTRUCTIONS 


Local Education Agencies (LEAs) in your btate 




provided Chapter 1 services? (ENTER 


Please complete this questionnaire for program 


NUMBER) (N=51) 


improvement activities occurring during school year 




(SY) 1990-91 or SY 1991-92, unless otherwise 


Median: 175 


specified. 


Range: 1 to 998 LEAs 


This questionnaire should be answered by you or 




the person you designate as most knowledgeable 


2. During SY 1990-91. how many schools in your 


about Chapter 1 program improvement activities in 


state provided Chapter 1 services? (ENTER 


your state. 


NUMBER) (N=50) 


Please complete and return this questionnaire in the 


Median: 759 


enclosed business reply envelope within 


Ranee: 88 to 3,880 Schools 


2 weeks of receipt to: 




Tim Silva 


3 During SY 1990-91, how many schools in your 


U.S. General Accounting Office 


state, if any, were in joint program 


Room 6737 


improvement? (ENTER NUMBER: IF NONE, 


441 G St.. NW 


ENTER *0') (N=51) 


Washington, DC 20548 






Total: 359 Schools (In 5 states, including 289 


If you have any questions, please call Tim Silva or 


in California, which does 


Richard Wcnmng, collect, at (202)401-8921. 


not have a local phase) 
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4. During SY 1990-91, how many schools in your 
state were in local program improvement? 
(ENTER NUMBER) (N=49) 

Median: 122 

Range: 9 to 696 Schools in local program 
improvement, SY 1990-91 



Of the schools that were in local program 
improvement during SY 1990-91, (1) how 
many were in their first year of the local phase 
and (2) how many were in their second year of 
the local phase? (ENTER NUMBER; IF 
NONE, ENTER '0') (N=46) 

Median: 96 

(1) Range: 3 to 438 Number in their first year 
of local phase, SY 1990- 
91 



Median: 18 
(2) Range: 0 to 292 



Number in their 
second year of local 
phase, SY 1990-91 



IDENTIFICATION OF SCHOOLS FOR JOINT 
PROGRAM IMPROVEMENT 

6. Has your state completed the identification of 
all schools that will be in joint program 
improvement during SY 1991-92? (CHECK 
ONE) (N=50) 



1. [32] Yes 

2. [18] No- 



About when will your state 
complete identification of 
schools for joint program 
improvement? (ENTER 
DATE) 



Of the schools your SEA has identified, thus 
far, to begin joint program improvement during 
SY 1991-92, (1) how many are beginning the 
joint phase after one year in the local phase and 
(2) how many are beginning the joint phase 
after two years in the local phase? (ENTER 
NUMBER; IF NONE, ENTER *0*) (N=49) 

Median: 0 

(1) Range: 0 to 287 Number beginning joint 

phase after one year in 
the local phase 

Median: 3 

(2) Range: 0 to 101 Number beginning joint 

phase after two years in 
the local phase 



9. Has your state exempted any schools from joint 
program improvement for SY 1991-92? 
(CHECK ONE) (N=51) 



1. [ 6] Yes 

2, [45] No 



(GO TO QUESTION 10) 

(GO TO PAGE 3, 
QUESTION 12) 



10. How many schools were exempted from joint 
program improvement for SY 1991-92? 
(ENTER NUMBER) (N=5) 

Median: 2 

Ranee: 1 to 3 Schools 



"mc/ DA YR 



How many schools has your SEA identified, 
thus far, to begin joint program improvement 
during SY Wl-92? (ENTER NUMBER: IF 
NONE, ENfER'0') (N=51) 

Median: 7 

Range: 0 to 287 Schools in joint program 
improvement, SY 1991-92 
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1 1. What were the reasons that schools were 


lO TC A KTV CPUr\T»T Q HAVE TIPUM 
12. IT AJN X ol. tl vXJLiO flAVtl OCSIN 


exempted from joint program improvement 


TTHThrn prcn *m tits tm totmt pr ot?p a m 


during SY 1991-92? (CHECK ALL THAT 


rMPROVFMENT DURING SY 1991.02 


APPLY) 


(N=5) 


pn?MTTMTTP TO OTTF^TrONI 1 *V TP NH 






vciir\/~\f Q U A VP RPPM TDPTsTTTPTPn (TO 
o\^ti\jsJLiO n/vvc dccin Lumrt i ltuzlj, 


LEU 


The mobility cf the student population 


to pa of fk nirF^TrnN 1 17 


2. [ 1] 


The extent of educational deprivation 






among participating children 


13. Of all the schoolr identified to be in joint 






program improvement during SY 1991-92, how 


3. [0] 


The difficulties involved in dealing 


many, if any, were identified for each of the 




with older children in Chapter 1 


frtllfMifinn r*ac«nc9 fP\TT"PD KJTTTLfP.PD* TT* 
IOllOWing rCoSOns : ^Ca~ 1 CX\ ri U ivioLvXv, IT 




programs in secondary schools 


VJAVTC PWTPP. ' /7J— /i<"\ 


4. [2] 


Indicators other than improved 


mtttlchcd 




achievement demonstrated the positive 






effects of Chapter 1 on participating 


IWCUlaT . O 




children 


1. Range. 0 to 165 Number of schools 






identified only because of 


5. [ 2] 


A change in the review cycle. 


insufficient NCE gains 




measurement instrument used, or other 






measurement problems rendered the 






results invalid or unreliable 


9 Ranof 0 tn 9fi Mumlvr nf crhnnlc 






iHf»nfifii»H OTllv Jv^aiic^ of 
iuciiiiiivvi mil t UvvaUoc \fi 


6. [3] 


Other reason(s) (PLEASE SPECIFY) 


lack of substantial 






progress toward other 












Mfviiatv 0 






3. Range: 0 to 153 Number of schools 






identified because of both 






insufficient NCE gains 






and lack of substantial 






progress toward other 






desired outcomes 






14. As of February 1, 1992, about how many 






schools in your state, if any, will be fully 






implementing a joint program improvement 






plan? (ENTER NUMBER; IF NONE. 






ENTER '0*) (N=48) 






Mtdian: 1 






Rtnge: 0 to 289 Schools fully implementing a 






joint program improvement 






plan 
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15. Consider the schools in your state that will be in joint program improvement during SY 1991-92. 

In your opinion, to what extent, if any, did each of the following factors hinder these schools' ability to 
succeed during local program improvement? 

(CHECK ONE FOR EACH FACTOR) 



FACTORS THAT MAY HAVE 
HINDERED SCHOOLS' ABILITY 
TO SUCCEED 


Little or 
Mo Extent 
0) 


Some 
Extent 
(2) 


Moderate 
Extent 

(3) 


Great 
Extent 
(4) 


Very 
Great 
Extent 
(5) 


Don't 
know 
(6) 


1. Availability of funds for program 
improvement 


27 


11 


6 


0 


0 


1 


2. Availability of technical 
assistance from SEA 


27 


9 


6 


0 


2 


1 


3. Availability of technical 
assistance from TAC/RTAC 


32 


6 


4 


0 


- 


2 


4. Attitudes of LEA staff toward 
program improvement 


5 


11 


14 


10 


4 


1 


5. Attitudes of school staff toward 
program improvement 


6 


7 


13 


12 


4 


3 


6. Coordination between Chapter 1 
and the regular program 


3 


9 


li 


13 


7 


2 


7. Quality of regular classroom 
instruction 


3 


6 


15 


10 


3 


8 


8. Poverty among children served 


10 


13 


9 


7 


3 


2 


9. Educational deprivation among 
children served 


7 


10 


9 


13 


4 


2 


10. Having children in Chapter 1 
who would have been better 
served in special education 


15 


8 


5 


9 


0 


8 


1 1 . Delivery models used for 
Chapter 1 instruction 


12 


7 


14 


9 


2 


1 


12. Quality of Chapter 1 instructors 


11 


11 


9 


8 


3 


3 


13. Time allowed by law to show 
improvement 


15 


13 


8 


7 


1 


1 


14. Strategics selected to improve 
programs under local 
improvement plans 


6 


11 


11 


10 


5 


2 


1 5 Other factor (PLEASE 
SPECIFY) 

I 


0 


0 


2 


10 


4 


0 



(N=45) 

(N=45) 

(N=45) 

(N=45) 

(N=45) 

(N=45) 

(N=45) 
(N=44) 

(N=45) 

(N=45) 

(N=45) 
(N=45) 

(N=45) 

(N=45) 

(N=16) 
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16. Again, consider the schools in your state that will be in joint program improvement during SY 1991-92. In 
your opinion, to what extent, if any, will each of the following factors hinder these schools* ability to 
succeed during joint program improvement? 



(CHECK ONE FOR EACH FACTOR) 



FACTORS THAT MAY HINDER 
SUCCEED 


Little or 
no extern 


Some 
Extent 

(2) 


Moderate 
Extent 

(3) 


Great 
Extent 
(4) 


Very 
Great 
Extent 
(5) 


Don't 
Know 
(6) 




1. Availability of funds for program 
improvement 


LI 


1 1 


L 


L 


U 


2. 


rs ,—AA\ 


2. Availability of technical 
assistance from SEA 


2A 


1 1 


A 

4 


2. 


1 


2. 


n>j—AA\ 

{Pi— tIJ 


3. Availability of technical 
assistance from TAC/RTAC 


31 


9 


1 


0 


1 


3 


(N=45) 


4. Attitudes of LEA staff toward 
program improvement 


1U 


Id 


I 


O 
8 


4 


3 




5. Attitudes of school staff toward 
program improvement 


9 


13 


4 


11 


5 


3 


(N=45) 


6. Coordination between Chapter 1 
and the regular program 


5 


10 


13 


10 


3 


4 


(N=45) 


7. Quality of regular classroom 
instruction 


2 


10 


13 


8 


3 


9 


(N=45) 


8. Poverty among children served 


14 


13 


7 


6 


2 


3 


(N=45) 


9. Educational deprivation among 
children scrveu 


8 


11 


9 


12 


2 


3 


(N=45) 


10. Having children in Chapter 1 
who would be better served in 
special education 


20 


5 


8 


3 


1 


8 


(N=45) 


1 1 . Delivery models used for Chapter 
1 instruction 


10 


12 


13 


6 


1 


3 


(N=45) 


12. Quality of Chapter 1 instructors 


12 


13 


10 


3 


3 


4 


(N=45) 


13. Strategies selected to improve 
programs under joint 
improvement plans 


13 


9 


8 


5 


3 


7 


(N=45) 


14. Other factor (PLEASE 
SPECIFY) 


0 


2 


2 


5 


3 


0 


(N=12) 
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SEA ASSISTANCE TO SCHOOLS IN JOINT 
PROGRAM IMPROVEMENT 

17. Consider the kinds of assistance your SEA will 
provide to schools in joint program 
improvement compared to schools in local 
program improvement during SY 1991-92. 

Will your SEA provide any kinds of assistance 
to schools in joint program improvement that it 
will not provide to schools in local program 
improvement? (CHECK ONE) (N=50) 



1- [40] 


Yes — > (GO TO QUESTION 18) 


2- [6] 


No 


3.E 1] 


Don't know (GO TO 




QUESTION 19) 


4. [3] 


Not applicable - 




no schools will 




be in joint program 




improvement 



STATE STANDARDS FOR IDENTIFYING 
SCHOOLS 

19. During SY 1990*91. what was the minimum 
state NCE standard used to identify schools for 
Chapter 1 program improvement? (CHECK 



ONE) 




1. [28] 


Greater than 0 NCEs 


2. [15] 


1 NCE or greater 


3. [3] 


2 NCEs or greater 


4- [1] 


3 NCEs or greater 


5. [4] 


Other NCE standard (PLEASE 




SPECIFY) 



18. Below, please briefly describe up to three kinds 
of assistance your SEA will provide to schools 
in joint program improvement that it will not 
provide to schools in local program 
improvement. 

1. N=40 



20. For the purpose of identifying schools for 
program improvement during SY 1990-91, did 
your SEA require the use of desired outcomes 
other than NCEs in grades 2-12? (CHECK 
ONE) (N=5I) 

1 . [20] Yes. required 

2. [31] No, not required 



21. Consider a school that does not show enough 
of an NCE gain, but which docs have other 
evidence of the effectiveness of its Chapter 1 
2. N=33 program. 

In your state, could such schools be exempted 
from local or joint program improvement? 
(CHECK ONE) (N=51) 

1. [14] Yes. from both local and joint program 
improvement 

3 2. [ 6] Yes. from local program improvement 

pnlv, 

3. ( 0] Yes, from joint program improvement 

only. 

4. (31] No. not from local or joint program 

improvement 
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STATE VIEWS ON THE IDENTIFICATION PROCESS 

22. In your opinion, how accurate or inaccurate is the process in your state for identifying schools that need Chapter 1 
program improvement? (CHECK ONE) (N=51) 



I. 


[ 1] 


Always accurate — > (GO TO PAGE 8, QUESTION 28) 


2. 


[151 


Almost always accurate - — ■ 




3. 


[19] 


More often accurate than inaccurate 




4. 


[14] 


As often accurate as inaccurate 


-> (GO TO QUESTION 23) 


5. 


[2] 


More often inaccurate than accurate 




6. 


[0] 


Almost always inaccurate 




7. 


[0] 


Always inaccurate 





23. In your opinion, to what extent, if at all. is each of the following a reason for inaccuracies in your state's 
identification process? 



(CHECK ONE FOR EACH REASON) 



REASONS FOR INACCURACY 


Little or 

No 
Extent 

(1) 


Some 
Extent 
(2) 


Moderate 
Extent 
(3) 


Great 
Extent 
(4) 


Very 
Great 
Extent 
(5) 




I. School identification process is 
based on only one year of data 


6 


13 


9 


13 


8 


(N=49) 


2. Norm-referenced tests do not 
indicate Chapter 1 program 
effectiveness 


3 


16 


13 


11 


6 


(N=49) 


3. Schools arc identified if they fail 
to meet any single outcome (for 
example, NCEs cr another desired 
outcome) 


15 


9 


8 


9 


3 


(N=49) 


4. The NCE scores of a few students 
can put a school in or out of 
program improvement 


2 


11 


10 


12 


13 


(N=48) 


5. Other reason (PLEASE SPECIFY) 


0 


2 


3 


4 


4 


(N=13) 
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24. Consider the process for determining whether 
or not schools should be in the local phase of 
program improvement during SY 1991-92. 

Were any schools identified for the local phase 
that, in your opinion, do not need Chapter 1 
program improvement? (CHECK ONE) (N=48) 

1. (191 Yes 

2. [191 No 

3. [10] Don't know 



27. Again, consider the process for determining 
whether or not schools should be in the joint 
phase of program improvement during 
SY 1991-92. 

Were any schools not identified for the joint 
phase that, in your opinion, do need Chapter 1 
program improvement? (CHECK ONE) 

(N=50) 

1. [14] Yes 

2. [181 No 

3. [14] Don't know 



25. Again, consider the process for determining 
whether or not schools should be in the local 
phase of program improvement during SY 
1991-92. 

Were any schools not identified for the local 
phase that, in your opinion, do need Chapter 1 
program improvement? (CHECK ONE) 

(N=49) 

1. [24] Yes 

2. [ 8] No 

3. {17] Don't know 



4. [ 4] Not applicable- no schools identified 
to be in joint program improvement 



REQUEST FOR DOCUMENTS 

28. As part of our study, we also plan to survey 
LEA Chapter 1 coordinators and principals of 
schools involved in the joint program 
improvement process. To help us with our 
future data collection efforts, please enclose: 

1. A list of all LEAs in your state with one or 
more schools in joint program 
improvement, including: 



26. Now, consider the process for determining 
whether or not schools should be in the joint 
phase of program improvement during 
SY 1991-92. 

Were any schools identified for the joint phase 
that, in your opinion, do not need Chapter 1 
program improvement? (CHECK ONE) (N=49) 

1. [10] Yes 

2. [30] No 

3. [ 6] Don't know 

4. ( 3] Not applicable- no schools identified 

to be in joint improvement 



Name of district 

Name of Chapter 1 Coordinator 

Address 

Phone number 

Number of joint plan schools 

2. A list of all schools in your state in joint 
program improvement, including: 

Name of school 
Name of principal 
Address 
Name of LEA 



3. A copy of your SY 1990-91 SEA plan for 
implementing the Chapter 1 program 
improvement provision. 
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COMMENTS 

29. Below, please briefly discuss any concerns or 
recommendations you have about Chapter 1 
program improvement (N=5I) 

36 states provided comments 



30. Please provide below any comments you have 
about this questionnaire, or any of the 
questions. (N=51) 

16 states provided comments 
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U.S. GENERAL ACCOUNTING OFFICE 
JOINT PROGRAM IMPROVEMENT STUDY 
Survey of School District Chapter 1 Coordinator* 



INTRODUCTION 

The U.S. General Accounting Office, an agency of 
the Congress, is conducting a siudy of stale and 
local implementation of the Chapter 1 program 
improvement provision. As part of our study, we 
are sending this questionnaire to all school districts 
in the nation that have one or more schools in the 
joint phase of program improvement during school 
year(SY) 1991-92. 

INSTRUCTIONS 

This questionnaire asks about schools* experiences 
in both the local and joint phases of Chapter 1 
program improvement. These two phases are 
defined as follows: 

-- Local phase of program improvement: 

When a Chapter 1 school is initially identified 
as needing improvement, it enters the first, or 
local, phase of program improvement. During 
the local phase, the local education agency 
(LEA) is responsible for developing and 
implementing a local program improvement 
plan in coordination with the school. In some 
places, the first school year is spent developing 
a local plan which is implemented the 
following year; in other places, implementation 
begins during the first year. 

-- Joint phase of program improvement: A 

school enters the second, or joint, phase of 
program improvement if it is identified as still 
needing improvement after its local plan has 
been in effect for one full school year. During 
the joint phase, the LEA and the state education 
agency (SEA) must together develop and 
implement a joint program improvement plan in 
coordination with the school. Here again, in 
some places, one school year is spent 
developing a joint plan which is implemented 
the following year; in other places, 
implementation begins during the first year. 

This questionnaire should be answered by you or 
the staff member you designate as most 
knowledgeable about Chapter 1 program 
improvement activities in your distiict. 



Please complete and return this questionnaire in the 
enclosed business reply envelope within 10 days of 
receipt to: 

Tim Silva 

U.S. General Accounting Office 
NGB/Education & Employment Issues 
441 G Street. NW 
Washington. DC 20548 

If you have any questions, please call Tim Silva at 
(202) 512-7041. or Richard Wenning at (202) 512- 
7072. 

Please provide the name of the primary person 
responsible for completing this questionnaire so that 
we may call and clarify information, if necessary. 

Name: 

Title: . 

Office Phone: 

School District: 

State: 



1 . Arc any public or private schools in your 
school district currently in joint program 
improvement? (CHECK ONE) (N=478) 

* 1 . [83] Yes. public school(s) only 

2. [ 0] Yes. private school (s) only 

3. t 51 Yes. both public and private schools 

4. (131 No > (STOP! PLEASE 

RETURN THIS 
QUESTIONNAIRE. 
THANKS FOR YOUR 
HELPt) 

* Data presented using percentages, rounded to 
nearest whole number; totals may not sum to 
100 due to rounding. 
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SEA TECHNICAL ASSISTANCE FOR LOCAL 
PROGRAM IMPROVEMENT 

2. Consider the school(s) that are currently in the 

joint phase of program improvement. Did your district 
receive any technical assistance from the state education 

agency (SEA) when these schools were in the local phase? (CHECK ONE) (N=416) 

1. (86] Yes (GO TO QUESTION 3) 

2. (12) No (GO TO QUESTION 5) 

3. [ 3) Don't know (GO TO QUESTION 6) 



3. How much, if any, of each of the following types of technical assistance did your district receive from the SEA 
when these schools were in the local phase? 



(CHECK ONE FOR EACH TYPE OF TECHNICAL ASSISTANCE) 



TYPES OF SEA TECHNICAL 
ASSISTANCE 


Very 
Great 
Amount 
(1) 


Great 
Amount 
(2) 


Moderate 
Amount 
(3) 


Small 
Amount 
(4) 


Very 
Small 
Amount 
(5) 


No 
Amount 
(6) 




1. Explained the requirements of 
the local phase, in general 


18 


46 


29 


5 


2 


1 


(N=345) 


2. Provided general information 
about improving schools 
during the local phase 


14 


40 


35 


7 


3 


1 


(N=346) 


3. Provided specific assistance on 
improving individual schools 
during the local phase 


11 


23 


33 


16 


10 


8 


(N=342) 


4. Other (SPECIFY) 


0 


62 


25 


12 


0 


0 


(N=8) 



















4. Overall, how satisfied or dissatisfied were you 
with the tvpes of technical assistance your 
district received from the SEA when these 
schools were in the local phase? (CHECK 
ONE) (N=351) 

1. (32] Very satisfied 

2. |50| Generally satisfied 

3. [13] About as satisfied as dissatisfied 
4 [ 4 1 Generally dissatisfied 

5. [ 0] Very dissatisfied 



5. Overall, how satisfied or dissatisfied were you 
with the amount of technical assistance your 
district received from the SEA when these 
schools were in the local phase? (CHECK 
ONE) (N=393) 

1 . [28] Very satisfied 

2. [501 Generally satisfied 

3. [15] About as satisfied as dissatisfied 

4. ( 5] Generally dissatisfied 

5. [ 2] Very dissatisfied 
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LOCAL PROGRAM IMPROVEMENT ACTIVITIES 

6. Consider the school(s) that are now in the joint phase of program improvement. How much influence, if any, 
did each of the following sources have in determining what needed improvement in these schools during the 
local phase? 

(CHECK ONE FOR EACH SOURCE) 



SOURCES 


Very 
Great 
Influence 
(1) 


Great 
Influence 
(2) 


Moderate 
Influence 
(3) 


Some 
Influence 
(4) 


Little or 

No 
Infljence 

(5) 


1. Staff from these schools 


38 


40 


15 


5 


2 


2. Parents of students in these 
schools 


5 


19 


40 


25 


10 


3. Staff from other schools or 
districts 


2 


4 


12 


23 


59 


4. LEA staff 


26 


46 


20 


6 


2 


5. SEA staff 


7 


28 


29 


21 


14 


6. Chapter 1 Technical 

Assistance Center (TAC) or 
Rural-TAC staff 


4 


9 


17 


21 


49 


7. Other sources (SPECIFY) 


31 


33 


11 


6 


19 



(N=410) 

(N=411) 

(N=402) 
(N=409) 
(N=406) 

(N=397) 
(N=36) 
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Again, consider the school(s) that are now in the joint phase of program improvement. How much influence, if 
any, did each of the following factors have on the selection of improvement strategies for these schools during 
the local phase? 

(CHECK ONE FOR EACH FACTOR) 



FACTORS 


Very Great 
Influence 
(1) 


Great 
Influence 
(2) 


Moderate 
Influence 

(3) 


Some 
Influence 
(4) 


Little or 

No 
Influence 

(5) 


I. Staff from these schools 


38 


42 


12 


6 


2 


2. Parents of students in 
these schools 


6 


22 


35 


25 


12 


3. Staff from other schools 
or districts 


1 


5 


14 


21 


59 


4. LEA staff 


26 


45 


22 


5 


2 


5. SEA staff 


8 


26 


30 


21 


14 


6. Chapter 1 TAC or 
R-TAC staff 


4 


9 


16 


22 


49 


7. Literature on effective 
practices in schools 


15 


33 


30 


17 


5 


8. The need to show 
aggregate gains on 
norm -re fere need tests 


40 


38 


16 


4 


2 


9. Other factors 
(SPECIFY) 


46 


27 


8 


0 


19 



(N=411) 

(N=410) 

(N=401) 
(N=408) 
(N=407) 

(N=394) 

(N=406) 

(N=409) 

(N=26) 
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DISTRICT VIEWS OF LOCAL PROGRAM IMPROVEMENT EFFORTS 

8 Consider the schoot(s) in your district that are currently in the jojnt phase of program improvement. In your 
opinion, to what extent, if any, did each of the following factors hinder these schools' ability to succeed during 
the local phase of program improvement? 

(CHECK ONE FOR EACH FACTOR) 



FACTORS THAT MAY HAVE 
HINDERED SCHOOLS' ABILITY 
TO SUCCEED 


Very 
Great 
Extent 
(1) 


Great 
Extent 
(2) 


vtodcrate 
Extent 
(3) 


Some 
Extent I 
(4) 


Little or 
slo Extent 
(5) 


Don't 
Know 
(6) 




1 . Availability of technical 
assistance from SEA 


i 


3 


8 


13 


70 


5 


(N=406) 


2. Availability of funds for program 
improvement 


3 


12 


13 


16 


<i 


■5 
J 


(N=409) 


3. Attitudes of SEA staff toward 
program improvement 


2 


2 


5 


5 


5/ 


C 


(N=407) 


4. Attitudes of school staff toward 
program improvement 


4 


10 


20 


30 


34 


2 


(N=410) 


5. Coordination between Chapter 1 
and the regular program 


1 


!0 


21 


23 


43 


1 


(N=410) 


6 Quality of regular classroom 
instruction 


5 


11 


21 


27 


32 


3 


(N=408) 


7. Poverty among children served 


14 


17 


21 


22 


24 


2 


(N=408) 


8. Educational deprivation among 
children served 


17 


26 


17 


22 


16 


2 


(N=409) 


9. Mobility among children served 


12 


18 


18 


19 


31 


2 


(N=409) 


10. Degree of parental involvement 


14 


26 


24 


22 


13 


1 


(N=412) 


1 1 . Having children in Chapter I 
who would have been better 


10 


8 


14 


22 


44 


2 


(N=409) 


served in special education 
















12. Location of Chapter 1 instruction 
(such as pull-out, in-class, stand- 


I 


6 


14 


19 


59 


2 


(N^409) 


j alone, or others) 
















i — . 

1 13 Strategics used to improve 
programs in local phase 


2 


7 


19 


27 


42 


3 


(N=408) 


14. Quality of Chapter 1 teachers 


4 


6 


11 


19 


58 


2 


(N=400) 


15. Quality of Chapter 1 aides (para- 
prok- ,.->nals) 


*> 


6 


7 


19 


63 


4 


(N=382) 


16 Time allowed by law to show 
improve men i 


It 


19 


20 


19 


27 


4 


(N=409> 


17 Other factors (SPECIFY) 


52 


30 


10 


2 


3 


3 


(N=6U 



in 
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SEA TECHNICAL ASSISTANCE FOR JOINT 
PROGRAM IMPROVEMENT 

9. Thus far, has your district received any 

technical assistance from the SEA concerning 
the loint phase of program improvement? 
(CHECK ONE) (N=408) 

1. [911 Yes (GO TO QUESTION 10) 

2 [ 8] No (GO TO QUESTION 12) 

3. [ 2] Don't know (GO TO QUESTION 13) 



How much, if any, of each of the following types of technical assistance has your district received from the SEA 
concerning the joint phase of program improvement? 



(CHECK ONE FOR EACH TYPE OF TECHNICAL ASSISTANCE) 



TYPES OF SEA TECHNICAL 
ASSISTANCE 


Very 
Great 
Amount 
(1) 


Great 
Amount 
(2) 


Moderate 
Amount 
(3) 


Small 
Amount 
(4) 


Very 
Small 
Amount 
(5) 


No 
Amount 
(6) 




1. Explained the requirements of 
the lomt phase, in general 


29 


42 


23 


4 


1 


0 


(N=367) 


2. Provided general information 
about improving schools 
during the joint phase 


24 


41 


27 


7 


2 


0 


(N=365) 


3 Provided specific assistance on 
improving individual schools 
during the joint phase 


18 


32 


28 


10 


9 


2 


(N=366) 


4 Other (SPECIFY) 


31 


38 


31 


0 


0 


0 


(N=I3) 




















11. Overall, how satisfied or dissatisfied arc you 
with the types of technical assistance your 
district has received from the SEA concerning 
the joint phase? (CHECK ONE) (N=368) 

1. (38) Very satisfied 

2. (43) Generally satisfied 

3 [16] About as satisfied as dissatisfied 

4 I 3| Generally dissatisfied 
5. ( 1] Very dissatisfied 



12. Overall, how satisfied or dissatisfied arc you 
with the amount of technical assistance your 
district has received from the SEA concerning 
the joint phase? (CHECK ONE) (N=394) 

1. [32] Very satisfied 

2. [41] Generally satisfied 

3. [18] About as satisfied as dissatisfied 

4. [ 6] Generally dissatisfied 

5. [ 2} Very dissatisfied 
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JOINT PROGRAM IMPROVEMENT 
ACTIVITIES 

13. In general, what is the current status of your 
district's joint program improvement plans? 
(CHECK ONE) (N=403) 

1. [ 6] Joint plan development not yet begun 

(GO TO PAGE 9, QUESTION 16) 

2. [16] Joint plans being developed, but not 

completed 

3. Ill] Joint plans completed, but not 

implemented 

4. [38J Joint plans partially implemented 

5. [30] Joint plans fully implemented 



14. Thus far, how much influence, if any, have each of the following sources had in determining what needs 
improvement in the schools now in the joint phase? 



(CHECK ONE FOR EACH SOURCE) 



SOURCES 


Very 
Great 
Influence 
(1) 


Great 
Influence 
(2) 


Moderate 
Influence 
(3) 


Some 
Influence 
(4) 


Little or 

No 
Influence 

(5) 




I . Staff from these schools 


40 


38 


16 


4 


2 


(N=380) 


2. Parents of students in these 
schools 


5 


18 


41 


28 


8 


(N=380) 


3. Staff from other schools or 
districts 


2 


4 


12 


24 


58 


(N=376) 


4. LEA staff 


26 


46 


20 


7 


2 


(N=376) 


5. SEA staff 


9 


31 


30 


22 


9 


(N=379) 


6. Chapter I TAC or R-TAC 
?'.aff 


3 


8 


16 


22 


51 


(N=358) 


7. Other sources (SPECIFY) 


47 


13 


17 


7 


17 


(N=30) 












1 
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15, Thus far. how much influence, if any, have each of the following factors had on the selection of improvement 
strategies for the schools now in the joint phase? 

(CHECK ONE FOR EACH FACTOR) 



FACTORS 


Very Great 
Influence 
(I) 


Great 
Influence 
(2) 


Moderate 
Influence 
(3) 


Some 
Influence 
(4) 


muc or 

No 
Influence 

(5) 


I. Staff from these schools 


41 


38 


14 


4 


3 


2. Parents of students in 
these schools 


6 


18 


36 


31 


9 


3. Staff from other schools 
or districts 


1 


6 


11 


26 


57 


4. LEA staff 


27 


45 


19 


7 


2 


5. SEA staff 


9 


31 


29 


22 


9 


6. Chapter 1 TAC or 
R-TAC staff 


4 


10 


16 


21 


49 


7. Literature on effective 
practices in schools 


14 


35 


32 


15 


3 


8. The need to show 
aggregate gains on 
norm-referenced tests 


43 


37 


15 


4 


1 


9. Other factors 
(SPECIFY) 


52 


29 


10 


0 


10 



(N=383» 

(N=383) 

(N=377) 
(N=380) 
(N=381) 

(N=362) 

(N=378) 

(N=383) 

(N=21) 
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STANDARDS FOR IDENTIFYING SCHOOLS 



16. Last year (SY 1990-91), whai was the NCE 
standard used in your district to identify 
schools to be in program improvement? 
(CHECK ONE) (N=413) 

1. [25] ONCEsorless 

2. [22] Less than 1 NCE 

3. [27] Less than 2 NCEs 

4. [20] Less than 3 NCEs 

5. [ 3] Less than 4 NCEs 

6. [ 3] Other NCE standard (PLEASE 

SPECIFY) 



18. Last year (SY 1990-91), what types of desired 
outcomes, other than average NCE gains, were 
established for grades 2-12 to identify schools 
for program improvement? (CHECK ALL 
THAT APPLY) (N=239) 

1. [29] Attendance rates 

2. [14] Samples or portfolios of student work 

3. [36] Student performance on criterion- 

referenced test 

4. [52] Student performance on norm- 

referenced test 

5. [11] Length of time students remain in 

Chapter 1 program 



6. (50] Student grades 

7. [29] Retentions in grade 



17. Last year (SY 1990-91). for the purpose of 
identifying schools for program improvement, 
did your district require the use of desired 
outcomes in addition to the average NCE gains 
or grades 2-12? (CHECK ONE) 

(N=414) 



8. [ 9] Dropout rates 

9. [ 5] Graduation rates 

10. (31] Other (PLEASE SPECIFY) 



1. [59] Yes. required (GO TO 

QUESTION 18) 

2. [41] No. not required (GO TO 

QUESTION 19) 19. Consider a school that does not show enough 

of an NCE gain, but does have other evidence 
of the effectiveness of its Chapter 1 program. 



In your district, can such schools be exempted 
from local or joint program improvement? 
(CHECK ONE) (N=409) 



1. (7] Yes. from both local and joint program 

improvement 

2. [ 2] Yes, from local program improvement 

only 

3. [ 2] Yes, from joint program improvement 

only 

4. [75] No, not from local or join? program 

improvement 



5. (15] Don't know 



9 
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DISTRICT VIEWS ON THE IDENTIFICATION PROCESS 



20. In your opinion, how accurate or inaccurate is the process in your district for identifying schools that you believe 
need Chapter 1 program improvement? (CHECK ONE) (N=404) 

1. [17] Always accurate — > (GO TO PAGE 12, QUESTION 26) 

2. (19) Almost always accurate — 



3. [25] More often accurate than inaccurate 

4. [23] As often accurate as inaccurate 

5. [14] More often inaccurate than accurate 

6. [ 2] Almost always inaccurate 

7. [ 1] Always inaccurate — 



— > (GO TO QUESTION 21) 



21. i your opinion, to what extent, if at all, is each of the following a reason for inaccuracies in your district's 
identification process? 

(CHECK ONE FOR EACH REASON) 



REASONS FOR INACCURACY 


Very 
Great 
Extent 
(1) 


Great 
Extent 
(2) 


Moderate 
Extent 
(3) 


Some 
Extent 
(4) 


Linle 
or No 
Extent 
(5) 


1. School identification process is 
based on only one year of data 


26 


31 


15 


16 


13 


2. Norm -referenced tests do not 
indicate Chapter 1 program 
effectiveness 


38 


31 


15 


10 


6 


3. Schools are identified if they fail 
to meet the NCE standard, 
regardless of other evidence of 
their effectivenes . 


49 


30 


12 


6 


3 


4. The NCE scores of a few students 
can put a school in or out of 
program improvement 


60 


25 


9 


5 


1 


5. Other reason (PLEASE SPECIFY) 


80 


9 


9 


3 


0 







(N=322) 
(N=327) 

(N=331) 

(N=336) 
(N=35) 
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22. Consider the process for determining whether 
or not schools needed to be in the local phase 
of program improvement during SY 1991-92. 

Were any schools in your district identified for 
the local phase that, in your opinion, do not 
need to be in Chapter 1 program improvement? 
(CHECK ONE) (N=339) 

1 . [46] Yes -> About how many? Median: 2 

2. [44] No 

3. [ 7] Don't know 

4. [ 3) Not applicable - no schools assessed 

for local phase 

23. Again, consider the process for determining 
whether or not schools need to be in the local 
phase of program improvement during SY 
1991-92. 

Were any schools in your district not idenuficd 
for the local phase that, in >our opinion, do 
need to be in Chapter 1 program improvement? 
(CHECK ONE) (N=339) 

1. [10] Yes -> About how many? Median: 1 

2. [80] No 

3. [ 6] Don't know 

4. [ 3] Not applicable - no schools assessed 

for local phase 



24. Now, consider the process for determining 
whether or not schools need to be in the joint 
phase of program improvement during SY 
1991-92. 

Were any schools in your district identified for 
the joint phase that, in your opinion, do not 
need to be in Chapter 1 program improvement? 
(CHECK ONE) (N=343) 

1. [44] Yes — > About how many? Median: 1 

2. [50] No 

3. [ 6] Don't know 



25. Again, consider the process for determining 
whether or not schools need to be in the joint 
phase of program improvement during SY 
1991-92. 

Were any schools in your district not idenuficd 
for the joint phase that, in your opinion, do 
need to be in Chapter 1 program improvement? 
(CHECK ONE) (N=344) 

1. [ 7] Yes -> About how many? Median: 1 

2. [87] No 

3. [ 6] Don't know 
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DISTRICT AND SCHOOL BUILDING INFORMATION 

26. During SY 1990-91 and SY 1991-92, how much federal Chapter 1 funding, if any, was your district allocated 
from each of the following sources? (ENTER AMOUNT; IF NONE, ENTER *0') 



CHAPTER 1 ALLOCATION 


SY 1990-91 


SY 1991-92 


1 . Basic grant 


Median: $ 476,144 
(N=348) 


Median: $ 568,248 
(N=363) 


2, Concentration grant 


Median: S 44,226 
(N=268) 


Median: $ 65,724 
(N=284) 


3. Program improvement 
(sec. 1405) funds 


Median: $ 1,152 
(N=304) 


Median: $ 1,500 
(N=317) 



27. How many public and private schools in your district currently provide Chapter 1 services, and 

how many are in the local or joint phases of program improvement? (ENTER NUMBERS; IF NONE, 
ENTER '0') 





Public 
Schools 


Private 
Schools 


TOTAL 


I. Number of schools currently 
providing Chapter 1 services 


Median: 5 
(N=402) 


Median: 0 
(N=389) 


Median: 5 
(N=388) 


2. Number of schools currently in 
local program improvement 


Median: 1.5 
(N=392) 


Median: 0 
(N=38l) 


Median: 1.5 
(N=378) 


3. Number of schools currently in 
joint program improvement 


Median: 1 
(N=399) 


Median: 0 
(N=389) 


Median: 1 
(N=389) 



28. Arc any of the schools that are currently in 
joint program improvement officially 
designated as Chapter 1 School \. \ *c Projects? 
(CHECK ONE) (N=410) 

1. [19) Yes — > How many? Median: 1 

2. (78] No 

3. ( 2] Don't know 
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YOUR COMMENTS 

29. Below, please briefly discuss any concerns or 
recommendations you have about Chapter 1 
program improvement. (N=417) 

61% provided comments 



30. Please provide below any comments you have 
about this questionnaire, or any of the 
questions. (N=417) 

2 1 7c provided comments 



_ _ HRO/l.MM/4 \92 

THANKS FOR YOUR HELP! 
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U.S. GENERAL ACCOUNTING OFFICE 




JOINT PROGRAM IMPROVEMENT STUDY 




Surrey of Chapter 1 School Principals 




L±\ I Ix w 17 X lull 






i iic u.o. vjcncicu /\ccounung vjiiicc, <Ui agency 01 


find it helpful to consult with other staff members 


die Congress* is conducting a study of slate and 


when completing this survey. 


Iocs! implementation of the Chapter 1 program 






improvement provision. As part of our study* we 


Please complete and return this questionnaire in the 


axe sending this questionnaire to all schools in the 


enclosed business reply envelope within 10 days of 


nation that, according to their stats Chapter I 


receipt to: 




coordinators, are in the ioint phase of program 






improvement during school year (SY) 1991*92. 


Tun Silva 






U.S. General Accounting Office 


irxo 1 1\ u l i iui^ j 


NGB/Education & Employment Issues 




441 G Street. NW 




i "h ir flii^stmnnaire aclrc about vnur c/*hfv"it'c 


Washington. DC 20548 


cApviicuuo wiui Lnjui uic local axKi joint pnascs oi 






\_iiupici i prugioxn irnpruvcrncnu incsc i wo pnascs 


If you have any questions, please call Tim Silva at 


are defined as follows! 


(202) 512-7041, or Richard Wenning at (202) 512- 




7072. 




f 1 u t 

** Local phase of program improvement! 






♦men «i ^ iidpici i scnooi is initially lacnuiico 


Please provide the name of the primary person 


as needing improvement, it enters the first, or 


responsible for completing this questionnaire so 


im.di, pii<uc ut piugiarn irnprovcmcni. uunng 


that we may call and clarify information, if 


uic iuc«j pnuci uic ioc*i coucauon agency 


necessary. 




(\ FA^ rie<r>on^ihl<* fnr rf^v^Innino ar>H 






implementing a local program improvement 


Name: 




plan in coordination with the school. In some 






places, the first school year is spent developing 


Title: 




a local plan which is implemented the 






following vearl in c\thcr nlaces imnlrmrntatinn 


Office Phone: 




frv*o»n< Hnnno llv fircf vi/>ar 

UCglllA UUllllg 11 IV HIM j v4U . 








School: 




- Joint phase of program improvement: A 






school enters the second, or ioint. phase of 


School District: 




program improvement if it is identified as still 






needing improvement after its local plan has 


State- 




been in effect for one full school year. Dunng 






the joint phase, the LEA and the state education 


1. Has your school been identified lo be in the 


agency (SEA) must together develop and 


ioint phase of program 


improvement dunng 


implement a joint program improvement plan in 


school year (SY) 1991-92? (CHECK ONE) 


coordination with the school. Here again, in 




(N=1,199) 


some places, one school year is spent 


• 1.(77] Yes, identified 


developing a joint plan which is implemented 


(GO TO QUESTION 2) 


the following year: in other places. 






implementation begins dunng the first year. 


2. [ 11 Yes. identified 




but later 




This questionnaire should be answered by you or 


exempted 


(STOP! PLEASE 


the staff member you designate as most 




RETURN TlflS 


knowledgeable about Chapter 1 program 


3.(18] No. not 


- QUESTIONNAIRE, 


improvement acu vines in your school. You may 


identified 


THANKS FOR 






YOUR HELP!) 




4. ( 41 Don't know - 






* Data presented using percentages, rounded to 




nearest whole number: totals may not sum to 




100 due to rounding. 
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SEA TECHNICAL ASSISTANCE DURING LOCAL PROGRAM IMPROVEMENT 

2. Did your school receive any technical assistance 
directly from the state education agency (SEA) when 
your school was in the local phase of program 
improvement? (CHECK ONE) (N=907) 

1. (69} Yes (GO TO QUESTION 3) 

2. (231 No (GO TO QUESTION 5) 

3. ( 81 Don't know (GO TO QUESTION 6) 



3. How much, if any, of each of the following types of technical assistance did your school receive directly 
from the SEA when it was in the local pi aie of program improvement? 



(CHECK ONE FOR EACH TYPE OF TECHNICAL ASSISTANCE) 



TYPES OF SEA TECHNICAL 
ASSISTANCE 


Viry 
Great 
Amount 
(0 


Great 
Amount 
(2) 


Moderate 
Amount 
(3) 


Small 
Amount 
(4) 


Very 
Small 
Amount 
(5) 


No 
Amount 
(6) 




1. Explained the requirements 
of the local phase, in general 


13 


34 


38 


10 


5 


1 


(N=616) 


2. Provided genera] information 
about improving schools 
during the local phase 


12 


32 


39 


9 


7 


2 


(N=616) 


3. Provided specific assistance 
on improving your school 
during the local phase 


10 


22 


33 


16 


8 


12 


(N=609) 


4. Other (SPECIFY) 


25 


20 


35 


5 


5 


10 


(N=20) 



4. Overall, how satisfied or dissatisfied were you 
with the types of technical assistance your 
school received directly from the SEA when it 
was in the loca[ phase? (CHECK ONE) 

(N=617) 

1. (23] Very satisfied 

2. [471 Generally satisfied 

3. (23) About as satisfied as dissatisfied 

4. ( 6) Generally dissatisfied 

5. ( 1] Very dissatisfied 



5. Overall, how satisfied or dissatisfied were you 
with the amount of technical assistance your 
school received directly from the SEA when it 
was in the local phase? f CHECK ONE) 

(N=775) 

1. (171 Very satisfied 

2. [42] Generally satisfied 

3. (27] About as satisfied as dissatisfied 

4. [ 9] Generally dissatisfied 

5. [ 5] Very dissatisfied 
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Results of Survey of Chapter 1 
School Principals 



LOCAL PROGRAM IMPROVEMENT 
STRATEGIES 



6. Please indicate whether or not each of the following strategies was part of your school's local program 
improvement effort? 





(CHECK ONE FOR EACH 
STRATEGY) 




STRATEGIES 


Yes 
(1) 


No 
(2) 


Don't 
KiiOw 
(3) 




1. Adopting a new instructional approach (such as, 
cooperative learning) for Chapter 1 


66 


32 


2 


(N=899) 


2. Adopting a new instructional approach (such as, 
cooperative learning) for regular program 


56 


41 


2 


(N=889) 


3. Changing the curriculum taught to Chapter 1 students 


47 


52 


2 


(N=885) 


4. Changing the grade levels served in Chapter I program 


24 


75 


1 


(N=892) 


5. Changing the location of Chapter 1 services (such as, from 
pull-out to in-class or other location) 


42 


57 


1 


(N=900) 


6. Changing the student selection criteria for Chapter 1 
program 


20 


78 


2 


(N 897) 


7. Adding Chapter 1 services before or after school hours 
(extended day) 


21 


78 


1 


(N=904) 


8. Adding Chapter 1 services during the summer 


25 


73 


2 


(N=899) 


9. Increasing the number of Chapter 1 teachers 


24 


74 


2 


(N=893) 


10. increasing the number of Chapter 1 aides (para- 
professionals) 


26 


73 


I 


(N=884) 


1 1 . increasing parental involvement 


86 


12 


1 


(N=899) 


12. Improving coordination between Chapter 1 and the regular 
program 


92 


7 


1 


(N=902) 


13. Instructing Chapter 1 students on test-taking skills 


75 


23 


2 


(N=893) 


14. Other strategies (SPECIFY) 


95 


1 


4 


(N=157) 
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School Principals 



7. During the local phase, which of the following 
indicators of Chapter 1 program effectiveness 
were your improvement strategies intended to 
improve? (CHECK ALL THAT APPLY) 

(N=899) 

1. [48] Attendance rates 

2. [28] Samples or portfolios of student work 

3. [70] Student performance on criterion- 

referenced tests 

4. [94] Student performance on norm- 

referenced tests 

5. [30] Length of time students remain in 

Chapter 1 

6. [72] Student grades 

7. (42] Retentions in grade 

8. [23] Dropout rates 

9. [12] Graduation rates 

10. [12] Other indicators (SPECIFY) 



8. In your opinion, of all the indicators of 
program effectiveness that you checked in 
response to the previous question, which were 
the first, second, and third most important for 
your school to improve during the local phase? 

(N=878) 

(ENTER ONE ITEM NUMBER FROM 
QUESTION 7 FOR EACH IMPORTANCE 
LEVEL BELOW) 

I. #4 First most important 

2- #6 Sf7ond most important 

3- #3 Third most important 
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Results of Survey of Chapter 1 
School Principals 



SCHOOL VIEWS OF LOCAL PROGRAM IMPROVEMENT EFFORTS 

9. Consider your school's experience in the local phase of program improvement. In your opinion, to what extent, if 
any, did each of the following factors hinder your school's ability to succeed during the local phase? 

(CHECK ONE FOR EACH FACTOR) 



FACTORS THAT MAY HAVE 
HINDERED SCHOOL'S ABILITY 
TO SUCCEED 


Very 
Great 
Extent 
(U 


Great 
Extent 
(2) 


Moderate 
Extent 
(3) 


Some 
Extent 
(4) 


Little or 
No Extent 
(5) 


Don't 
Know 
(6) 


1 . Availability of technical 
assistance from SEA 


2 


5 


11 


14 


59 


8 


2. Availability of funds for program 
improvement 


8 


14 


16 


16 


43 


4 


3. Attitudes of SEA staff toward 
program improvement 


2 


L 


5 


7 


68 


18 


4. Attitudes of school staff toward 
program improvement 


2 


1 


14 


24 


51 


2 


5. Coordination between Chapter 1 
and the regular program 


2 


7 


15 


23 


52 


0 


6. Quality of regular classroom 
instruction 


3 


5 


12 


25 


53 


2 


7. Poverty among children served 


17 


22 


18 


20 


22 


2 


8. Educational deprivation among 
children served 


22 


25 


18 


18 


15 


2 


9. Mobilit* among children served 


14 


16 


18 


20 


30 


2 


10. Degre* of parental involvement 


21 


28 


25 


17 


7 


0 


11. Having children in Chapter 1 
who would have been better 
served in special education 


6 


9 


16 


22 


43 


4 


12. Location of Chapter I instruction 
(such as pull-out. in-class, stand- 
alone, or others) 


2 


7 


11 


18 


61 


2 


13 Strategics used to improve 
programs in local phase 


2 


5 


U 


24 


54 


2 


14. Quality of Chapter 1 teachers 


3 


4 


6 


15 


68 


4 


15 Quality of Chapter I aides (para- 
professtonals) 


2 


3 


5 


14 


69 


6 


16. Time allowed by law to show 
improvement 


13 


16 


18 


19 


31 


4 


17. Other factors (SPECIFY) 


55 


28 


5 


I 


5 


4 



(N=869) 

(N=884) 

(N=87l) 

(N=884) 

(N=877) 

(N=875) 
(N=884) 

(N=882) 
(N=88l) 
(N=894) 

(N=877) 

(N=876) 

(*N=864) 
(N=r845) 

(N=804) 

(N=880) 

(N=92) 
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Results of Survey of Chapter 1 
School Principals 



IDENTIFICATION FOR JOINT PHASE OF 
PROGRAM IMPROVEMENT 

In answering the following questions, please recall 
that the joint phase of program improvement 
involves both the local and state education agencies 
(LEA and SEA). 

10. On what basis was your school identified to be 
in joint program improvement during SY 1991- 
92? (CHECK ONE) (N=902) 

1. [68] School identified only because of 
insufficient NCE gains 



13. In your opinion, which of the following, if any, 
are reasons that your school was inaccurately 
identified for the joint phase of program 
improvement? (CHECK ALL THAT APPLY) 

(N-265) 

1. [63] Norm-re fcrcnccJ tcsu» do not indicate 

the effectiveness of our Chapter 1 
program 

2. J57] Our school un* identified because we 

failed to meet the NCE standard, 
regardless of other evidence of out 
effectiveness 



2. ( 5] School identified only because of lack 
of substantial progress toward other 
desired outcomes 



3. [541 Th e NCE scores* of a few smdents pui 
our school in joint program 
improvement 



3. [23] School identified because of both 4. [22] Other reason (PLEASE SPECIFY ) 

insufficient NCE gains and lack of 

substantial progress toward other 

desired outcomes 



4. [ 4] Don't know 

II. Is SY 1991-92 your school's first or second 
year in the toin i phase of program 
improvement? "(CHECK ONE) (N=895) 

1. [761 First year in joint phase 

2. [24] Second year in joint phase 

] 2. Now, consider the process that was used to 
determine that your school is in need of {pint 
program improvement during SY 199L92 



SEA TECHNICAL ASSISTANCE Of RING 
JOINT PROGRAM IMPROVEMENT 

14. Thus far, has your school received an> 
technical assistance directly from the SEA 
concerning the ioint phase of program 
improvement? (CHECK ONE' iN=W , 

1. [73] Yes (GO TO Ql'ESTfON 15) 

2. [22] No (GO TO QUESTION 17) 

3. I 5] Don't know (GO TO QUESTION 181 



In your opinion, was your school accurately 
identified as in need of Chapter 1 program 
improvement during SY 1991-92? (CHECK 
ONE) (N=900) 



1 [631 Yes --> (GO TO QUESTION 14) 

2. [30] No •-> (GO TO QUESTION 13) 

3 [ 7] Don't know --> (GO TO QUESTION 
14) 
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School Principals 



15. Thus far, how much, if any, of each of the following types of technical assistance has your school received directly 
from the SEA concerning the joint phase of program improvement? 



(CHECK ONE FOR EACH TYPE OF TECHNICAL ASSISTANCE) 



TYPES OF SEA TECHNICAL 
ASSISTANCE 


Very Great 
Amount 
(1) 


Great 
Amount 
(2) 


Moderate 
Amount 
(3) 


Small 
Amount 
(4) 


Very 
Small 
Amount 
(5) 


No 
Amount 
(6) 




1. Explained the requirements 
of the joint phase, in general 


18 


37 


29 


7 


8 


1 


(N=664) 


2. Provided general information 
about improving schools 
dunng the joint phase 


15 


34 


33 


7 


6 


5 


(N=664) 


3. Provided specific assistance 
on improving your school 
during the joint phase 


12 


28 


28 


13 


8 


10 


(N=664> 


4. Other (SPECIFY) 


24 


43 


14 


0 


10 


10 


(N=21) 



















16. Overall, how satisfied or dissatisfied are you 
with the types of technical assistance your 
school has received directly from the SEA 
concerning the joint phase? (CHECK ONE) 

(N=664) 



I. 


(23| 


Very satisfied 


2. 


147) 


Generally satisfied 


3. 


[221 


About as satisfied as dissatisfied 


4. 


[ 8] 


Generally dissatisfied 


5. 


[0] 


Very dissatisfied 



17. Overall, how satisfied or dissatisfied arc you 
with the amount of technical assistance your 
school has received concerning the ioint phase? 
(CHECK ONE) (N=844) 

1. [17] Very satisfied 

2. [39] Generally satisfied 

3. (27) About as satisfied as dissatisfied 

4. (11) Generally dissatisfied 

5. [ 5] Very dissatisfied 



STATUS OF JOINT PROGRAM 
IMPROVEMENT ACTIVITIES 

18. About when did (will) your school begin fullv 
implementing its joint program improvement 
plan? (ENTER MONTH AND YEAR) 

(N=896) 

/ [88] 

MONTH YEAR 

[12] Don't know 



19. What is the current status of your school's 
improvement plan for the joint phase? (CHECK 
ONE) (N=889) 

1. (4] Joint plan development not yet begun 

(GO TO PAGE 9, QUESTION 24) 

2. [12] Joint plan being developed, but not 

completed 

3. [11] Joint plan completed, but not 

implemented 

4. [37] Joint plan partially implemented 

5. [36] Joint plan fully implemented 
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JOINT PROGRAM IMPROVEMENT STRATEGIES 

20. Please indicate whether or not each of the following strategies are (will be) part of your school's joint program 
improvement effort? 





(CHECK ONE FOR EACH 
STRATEGY) 




STRATEGIES 


Yes 
(1) 


No 
(2) 


Don't 
Know 
(3) 




1 . Adopting a new instructional approach (such as, 
cooperative learning) for Chapter \ 


73 


23 


4 


(N=857) 


2. Adopting a new instructional approach (such as, 
cooperative learning} for regular program 


65 


30 


5 


(N=854) 


3. Changing the curriculum taught to Chapter 1 students 


45 


52 


3 


(N=344) 


4. Changing the grade levels served in Chapter 1 program 


25 


72 


3 


(N=839) 


5. Changing the location of Chapter 1 services (such as, from 
pull-out to in-class or other location) 


39 


58 


3 


(N-852) 


6. Changing the student selection criteria for Chapter 1 
program 


23 


73 


4 


(N=851) 


7. Adding Chapter 1 services before or after school hours 
(extended day) 


24 


72 


5 


(N=854) 


8. Adding Chapter 1 services during the summer 


28 


b7 


5 


(N=851) 


9 Increasing the number of Chapter 1 teachers 


27 


65 


8 


(N=849) 


10. Increasing the number of Chapter 1 aides (para- 
professionals) 


25 


69 


6 


(N=834) 


11. Increasing parental involvement 


94 


4 


2 


(N=859) 


12. Improving coordination between Chapter 1 and the regular 
program 


93 


6 


1 


(N=857) 


13. Instructing Chapter 1 students on test-taking skills 


83 


15 


2 


(N=855) 


14. Other strategies (SPECIFY) 


98 


0 


2 


(N=U0) 



! AVAILABLE 
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21. Which of the following indicators of Chapter 1 
program effectiveness are your improvement 
strategies during the joint phase intended to 
improve? (CHECK ALL THAT APPLY) 

(N=862) 

1. [51] Attendance rates 

2. [391 Samples or portfolios of student work 

3. [71] Student performance on criterion- 

referenced tests 

4. [94] Student performance on norm- 

referenced tests 

5. [35] Length of time students remain in 

Chapter 

6. [78] Student grades 

7. [50] Retentions in grade 

8. [24] Dropout rates 

9. [14] Graduation rates 

10. [ 14] Other indicators (SPECIFY) 



23. Overall, how similar, if at all, arc the 
improvement strategies that your school is 
adopting (or will adopt) in the joint phase to 
those adopted in the local phase? (CHECK 
ONE) (N=850) 

1. [17] Exactly the same 

2. [50] Very similar 

3. [19] Moderately similar 

4. [10] Somewhat similar 

5. [ 4] Not at all similar 

INFORMATION ABOUT YOUR SCHOOL 

24. Is this a public or private school? (CHECK 
ONE) (K-909) 

1. [98] Public 

2. [ 2] Private 



25. How many students are currently enrolled in 
your school? (ENTER NUMBER) (N=893) 

Median: 474 

Range: 15 to 2,900 Students in school 



22. In your opinion, of all the indicators of 
program effectiveness that you checked in 
response to the previous question, which are 
(will be) the first* second* and third most 
important for your school to improve during the 
joint phase? (N=838) 

(ENTER ONE ITEM NUMBER FROM 
QUESTION 21 FOR EACH IMPORTANCE 
LEVEL BELOW) 

1 *4 First most important 

2. *6 Second most important 

3 *3 Third most important 



26. How many students in your school arc 
currently receiving Chapter I services? 
(ENTER NUMBER) (N=866) 

Median: 128 

Range: 1 1 to 1.342 Students in Chapter 1 



27. How many students in your school are 
currently in the free lunch or reduced-price 
lunch program? (ENTER NUMBER; IF 
NONE, ENTER *0') 

Median: 240 

Range- 0 to 1.695 Students in free lunch 

program (N=833) 

Median: 29 

Range: 0 to 590 Students in reduced-price 
lunch program (N=794) 
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28. About what percentage of your students, if any, 
are in each of the following racial/ethnic 
categories? (ENTER PERCENTAGES; IF 
NONE, ENTER '0') (N=880) 



Mean: 42 


% White (not Hispanic) 


Mean: 


46 


% Black 


Mean: 


6 


% Hispanic (not black) 


Mean: 


2 


% Asian/Pacific Islander 


Mean: 


4 


% Native American/Alaskan 
Native 


Mean: 


0 


% Other/Unknown 



100% TOTAL 

29. During SY 1990-91 and SY 1991-92, how 
much program improvement funding, if any, 
was your school allocated? (ENTER 
AMOUNT; IF NONE, ENTER '0') 

$ Median: 0.00 SY 1990-91 (N=603) 

S Median: 0.00 SY 1991-92 (N=615) 



30. How many full-time and part-time Chapter 1 
teachers and aides (para .professionals), if any, 
arc currently employed at your school? 
(ENTER NUMBERS: IF NONE, 
ENTER '0') 



31. Below, please indicate which grades (1) are 
taught in your school and (2) are served by 
your Chapter 1 program. (CHECK ALL THAT 
APPLY FOR EACH GRADE) (N=904) 



GRADES 


Grades 
Taught 
in School 
(1) 


Grades 
Served by 
Chapter 1 
(2) 


1. Pre-K 


28 


8 


2. Kindergarten 


70 


33 


3. Grade 1 


73 


54 


4. Grade 2 


74 


61 


5. Grade 3 


75 


69 


6. Grade 4 


72 


64 


7. Grade 5 


70 


60 


8. Grade 6 


52 


43 


9. Grade 7 


33 


28 


10. Grade 8 


33 


27 


11. Grade 9 


14 


9 


12. Grade 10 V 


11 


6 


13. Grade 11 


11 


5 


14. Grade 12 


10 


4 





Number of 
Chapter 1 
Teachers 


Number of 
Chapter 1 
Aides (Para- 
profcssionals) 


1. Full-time 


Median: 2 
(N=860) 


Median: 2 
(N=850) 


2. Part-time 


Median: 0 
(N=784) 


Median: 0 
(N=773) 


TOTAL 


Median: 2 
(N=775) 


Median: 2 
(N=760) 
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YOUR COMMENTS 

32. Below, please briefly discuss any concerns or 
recommendations you have about Chapter 1 
program improvement (N=916) 

40% provided comments 



33. Please provide below any comments you have 
about this questionnaire, or any of the 
questions. (N=916) 

14% provided comments 



THANKS FOR YOUR HELP! 
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Case Study Locations 



a i West Memphis Public School District, 

ATKansaS Wedlock Elementary School 

Little Rock Public School District, 
Badgett Elementary School 



M 1 i Washington County School District, 

iViaryiana Hickory Elementary School 

Baltimore City Public Schools, 
Lexington Terrace Elementary School 



M . v . Hartford Public School District, 

MlClllgan Red Arrow Elementary School 

Detroit Public School District, 
Spain Elementary School 



TV/t* • • • Western Line School District, 

MISSISSIPPI Glen Elementary School 

Jackson City Public School District, 
Isable Elementary School 



Appendix VIII 

Comments From the 
Department of Education 



UNITED STATES DEPARTMENT OF EDUCATION 

OFFICE OF ELEMENTARY AND SECONDARY EDUCATION 

THE ASSISTANT SECRETARY 

Ms. Linda G. Morra 
Director, Education and 

Employment Issues 
Human Resource Division 

United States General Accounting Office 
Washington, DC 20548 

Dear Ms. Morra: 

The Secretary has asked that I respond to your request for 
comments on the GAO draft report, "CHAPTER 1 ACCOUNTABILITY: 
Greater Focus on Program Goals Needed" (GAO/HRD-93-69) , which was 
transmitted to the Department of Education by your letter of 
February 8, 199 3. 

Generally, we agree with the conclusions reached in the report 
and believe that they would be helpful to the Chapter l program 
operation. This general agreement, however, should be tempered 
with a few observations arising from the study design itself. 

We are in complete agreement with the conclusions that address 
the shortcomings of norm-referenced tests in identifying schools 
for program improvement and highlight the promise of multiple 
measures. There is no question that the use of multiple desired 
outcomes in identifying schools for program improvement would 
enhance the accuracy of the identification process. At the same 
time, however, it is necessary to recognize some limitations in 
the use of multiple measures. The first is the issue of data 
conflict; i.e., when one data source suggests program success and 
another data source suggests program failure. Department of 
Education monitors have observed norm-referenced test data may 
conflict with desired outcome data in local educational agencies 
that use multiple measures. This phenomenon may occur with any 
set of multiple measures. Although this is not an insolvable 
problem, it does require some sophistication in affecting a 
resolution. 

Second, the use of multiple measures will increase th« accuracy 
of the identification process only if the measures are valid and 
reliable. This should be noted in the report to emphasize the 
importance that measures used must be of high quality. Third, 
while we agree with your conclusions, as noted earlier, the study 
did not empirically verify the proposition that multiple measures 
are more precise than single measures in identifying schools for 
program improvement . 
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Page 2 - Ms. Linda G. Morra 

We are also enclosing some comments on portions of the draft 
report discussing the requirements for setting desired outcomes 
and aggregate achievement. We recommend that the final report 
reflect these changes. 

Thank you for the opportunity to comment. I and members of my 
staff are prepared to respond, if you or your representatives 
have any questions. 




Sincerely, 
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Additional Desired Outcomes Used by 
School Districts Involved in the Joint Phase 



Among school districts we surveyed that were involved in the joint phase 
during school year 1991-92, about 59 percent reported using other desired 
outcomes to identify schools for program improvement, in addition to 
average nce gains for grades 2 to 12. 1 The most commonly used type of 
desired outcome concerned student performance on achievement 
tests — using some measure other than the average annual gains of all 
Chapter 1 students. School districts in Illinois, for example, were required 
by the state to identify schools for program improvement if (1) less than 
75 percent of Chapter 1 students made gains on achievement tests, or 
(2) less than two-thirds of the grades served by Chapter 1 made average 
gains on achievement tests. The second most widely used desired outcome 
concerned students 1 grades, and the third concerned student performance 
on criterion-referenced tests. (See fig. IX. 1.) 



Figure IX.1: Types of Desired 
Outcomes, Other Than Average NCE 
Gains, Used by Districts Involved in 
the Joint Phase (N=239) 



60 Psrcsnt of Districts 
55 
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Typss of Dssired Outcomes 



! In addition, in school year lflOO -fit, for I ho purpose of identifying schools for program improvement, 
20 states required the use of desire* I outcomes, other than NCK Hains, for students in grades 2 to 12. 
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Local Conditions Specified in the Statute 



The local conditions section of the Hawkins-Stafford Amendments of 1988 
reads as follows: 

The local educational agency and the State educational agency, in perfoiming their 
responsibilities under this section, shall take into consideration— 

(1) the mobility of the student population, 

(2) the extent of educational deprivation among program participants which may 
negatively affect improvement efforts, 

(3) the difficulties involved in dealing with older children in <^ondary school programs 
funded under this chapter, 

(4) whether indicators other than improved achievement demonstrate the positive effects 
on participating children of the activities funded under this chapter, and 

(5) whether a change in the review cycle ... or in the measurement instrument used or othe 
measure-related phenomena has rendered results invalid or unreliable for that particular 
year. 1 



'Public Uw 100-207 section 1021(e). 
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School, District, and State Officials' Views 
on the Factors That Hindered Success in the 
Local Phase 



The questionnaires we sent to school, district, and state officials asked 
respondents for their opinions on the extent to which various factors 
hindered schools from succeeding daring the locai phase of program 
improvement 

Principals and district coordinators had generally similar views, ranking 
the same five factors at the top of the list. The factors they saw as having 
the greatest negative impact on improvement efforts were (1) degree of 
parental involvement, (2) educational deprivation among children served, 
(3) poverty among children served, (4) mobility among children served, 
and (5) time allowed by law to show improvement. 

The list of possible responses on the state coordinator questionnaire was 
somewhat different. Among factors that appeared on all three 
questionnaires, though, state coordinators sometimes had very different 
views from principals and district coordinators. For example, state 
coordinators were much more likely to say that schools were hindered in 
the local phase to a Very great" or "great" extent by (1) the coordination 
between Chapter 1 and the regular program, (2) the attitudes of school 
staff toward program improvement, and (3) the strategies used to improve 
programs in the local phase. (See table XI. 1.) 




1 it 

Pa ** 106 * GACVHRD-93-69 Chapter 1 Accountability 



Appendix XI 

School, District, and State Officials' Views 
on the Factors That Hindered Success in the 
Local Phase 



Table XI.1: Principals', District 
Coordinators', and State Coordinators' 
Opinions on Factors That Hindered 
School Success in the Local Phase 



Factors that may have hindered 
schools' ability to succeed in the 
local phase 


Percentage saying factor hindered schools* 
success to a "very great" or "great" extent 


Principals 


District 
coordinators 


State 
coordinators 


Degree of parental involvement 


50 


40 


a 


Educational deprivation among children 
served 


46 


44 


38 


Poverty among children served 


38 


31 


23 


Mobility among children served 


30 


30 


a 


Time allowed by law to show 
improvement 


29 


30 


18 


Availability of funds for program 
imnrovpmpnt 

III 'r-" ^ v \** 1 1 1 W 1 1 l 


22 


15 


0 


Having children in Chapter 1 who would 
have been better served in special 
education 


15 


18 


20 


Coordination between Chapter 1 and 
the regular program 


10 


11 


44 


Location of Chapter 1 instruction (such 
as pull-out, in-class, stand-alone, or 
other) 


9 


7 


a 


Attitudes of schooi staff toward program 
improvement 


9 


14 


36 


Quality of regular classroom instruction 


8 


16 


29 


Availability of technical assistance from 
SEA 


7 


5 


4 


Quality of Chapter 1 teachers 


7 


10 


a 


Strategies used to improve programs in 
local phase 


6 


9 


33 


Quality of Chapter 1 aides 
(para-professionals) 


6 


8 


a 


Attitudes of SEA staff toward program 
improvement 


3 


3 


a 


Attitudes of LEA staff toward program 
improvennent 


b 


b 


31 


Quality of Chapter 1 instructors 


b 


b 


24 


Delivery models used for Chapter 1 
instruction 


b 


b 


24 


Availability of technical assistance from 
TAC/R-TAC 


b 


b 


2 


(Table notes on next pag6) 
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Appendix XI 

School, District } and State Officiate' Views 
on the Factors That Hindered Success in the 
Local Phase 



Note: SEA = state education agency; LEA = local education agency; TAC = technical assistance 
center; R-TAC = rural technical assistance center. 

a ltem not included in the questionnaire for state coordinators. 

b ltem not included in the questionnaires for principals and district coordinators. 
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